基于SVM的词频统计中文分词研究

Study on Chinese word segmentation based on statistic and SVM

本文详细介绍SVM(支持向量机)在词频统计中文分词中的应用。可将输入的连续字串进行分词处理,输出分割后的汉语词串,一般为二字词串,并得到一个词典。词典中不重复地存储了每次处理中得到的词语,以及这些词语出现的频率。选用了互信息原理进行统计。并采用SVM算法,分词的准确性与传统相比有了很大的提高,并具有一定的稳定性。

The paper introduces the application of SVM in Chinese word segmentation, which is based on statistic the frequency of the word. Through the system, continuous character bunch input can be segmented, and then the cut apart word bunch output can be got ten, the cut apart word bunch usually is two character word bunch, and one dictionary can be gotten. The dictionary stores word and the frequency that the word appears in these disposal tests. The segmentation system selects Mutual Information to statistic. Use SVMt the veracity of segmentation was better than the traditional method, and is of high stability.