基于Hash结构的逆向最大匹配分词算法的改进

Improvement on reverse directional maximum matching method based on hash structure for Chinese word segmentation

分析中文的语义,首先要对句子进行分词。中文分词是中文信息处理中最重要的预处理,分词的速度和精度直接影响信息处理的结果。对传统的分词词典和算法进行了改进,提出了基于Hash结构的分词词典机制,并给出了一种改进的逆向最大匹配分词算法(RMM)。该算法在重点考虑切分速度的同时兼顾了切分精度,在很大程度上消除了传统的最大匹配算法可能产生的歧义。实验结果表明,该分词算法在运行效率和结果的准确性方法有了很大的提高。

To analyse the Chinese semantic phrases, one must divide the sentences into words. Chinese segmentation is the most important part of Chinese information process. The speed and accuracy of segmentation influence the results of information processing. Traditional dictionary mechanisms and word segmentation methods are improved. Meanwhile, a new dictionary mechanism is provided based on hash structure, and an improved reverse directional maximum match method （RMM） is put forward. This method emphasized particularly on the speed of segmentation and the accuracy of segmentation, and it largely dispelled some ambiguities that may be produced by traditional maximum matching method. The experiment indicates that the segmentation method is improved obviously on running efficiency and veracity of the results.