基于EM算法的汉语自动分词方法

Segmenting Chinese by EM Algorithm

汉语自动分词是中文信息处理中的基础课题。本文首先对汉语分词的基本概念与应用 ,以及汉语分词的基本方法进行了概述。接着引出一种根据词的出现概率、基于极大似然原则构建的汉语自动分词的零阶马尔可夫模型 ,并重点剖析了EM(Expectation Maximization)算法 ,对实验结果进行了分析。最后对算法进行了总结与讨论。

Word segmentation is a basic task of Chinese information processing.In this paper we present a simple probabilistic model of Chinese text based on the occurrence probability of the words,which can be seen as a zero-th order hidden Markov Model(HMM).Then we investigate how to discover by EM Algorithm the words and their probabilities from a corpus of unsegmented text without using a dictionary.The last part is conclusion and discussion about the algorithm.