基于虚词停顿的中文分词消歧研究

Research on the Sense Disambiguation of Chinese Segmentation Based on the Pause of the Form Word

提出一种基于虚词停顿的中文分词消岐的模型。首先利用建立的虚词知识库对文本进行粗分词-划分停顿,然后对句子中停顿间的短语用双向最大匹配再进行分词,提取歧义部分,最后使用N-Gram模型和数据平滑等技术处理。整个过程分为粗分词、精分词和歧义消除三个过程。测试结果显示,该模型能有效地降低词歧义引起的错误切分率。

This paper puts forward a model which can eliminate sense ambiguity of Chinese segmentation based on the pause of the empty words. Firstly, this model segments words roughly based on the empty words library and then it has many phrases between pauses. Secondly, they segment phrases based on MM and RMM and extract the ambiguity. Finally, they have the model of N-Gram and the technology of the data smoothing to improve it. The process can be divided into three parts ： segments word roughly, segments word nar rowly and disambiguation. The test result shows that this model is able to reduce the error rate of segmentation ,which is caused by the ambiguity of word segmentation.