基于自适应中文分词和近似SVM的文本分类算法

Text Classification Algorithm Based on Adaptive Chinese Word Segmentation and Proximal SVM

中文分词的难点在于处理歧义和识别未登录词,传统字典的匹配算法很大程度上是依靠字典的代表性而无法有效地识别新词,特别是对于各种行业领域的知识管理。基于二元统计模型的分词算法能很好地适应不同的语料信息,且时间和精度都能满足文本知识管理的应用需要。近似支持向量机是将问题归结成仅含线性等式约束的二次规划问题,该算法的时间复杂度和空间复杂度比传统SVM算法的均有降低。在利用自适应分词算法进行分词的基础上,再利用近似支持向量机进行文本分类。实验表明,该方法能够自动适应行业领域的知识管理,且满足文本知识管理对训练时间敏感和需要处理大量文本的苛刻环境要求,从而具备较大的实用价值。

New words recognition and ambiguity resolving are key problems in Chinese word segmentation. The result of traditional dictionary-based matching algorithm largely depends on the representative of the dictionary so that it can not recognize new words effectively, especially in some professional domains. Chinese word segmentation method in this dissertauon is based on 2-gram statistical model and can meet the requirements of application in accuracy and efficiency respectively. PSVM takes classification as a linear equality quadratic programming problem. This dissertation describes a text classification algorithm based on adaptive chinese word segmentation and PSVM, which has faster training speed and smaller memory requirements advantages. Several data sets of experiments showed that the classification algorithm can automatically adapt to knowledge management of some professional domains and has better elassfication performance under the condition of time-sensitive.