摘要
分析中文的语义,首先要对句子进行分词。中文分词是中文信息处理中最重要的预处理,分词的速度和精度直接影响信息处理的结果。对传统的分词词典和算法进行了改进,提出了基于Hash结构的分词词典机制,并给出了一种改进的逆向最大匹配分词算法(RMM)。该算法在重点考虑切分速度的同时兼顾了切分精度,在很大程度上消除了传统的最大匹配算法可能产生的歧义。实验结果表明,该分词算法在运行效率和结果的准确性方法有了很大的提高。
To analyse the Chinese semantic phrases, one must divide the sentences into words. Chinese segmentation is the most important part of Chinese information process. The speed and accuracy of segmentation influence the results of information processing. Traditional dictionary mechanisms and word segmentation methods are improved. Meanwhile, a new dictionary mechanism is provided based on hash structure, and an improved reverse directional maximum match method (RMM) is put forward. This method emphasized particularly on the speed of segmentation and the accuracy of segmentation, and it largely dispelled some ambiguities that may be produced by traditional maximum matching method. The experiment indicates that the segmentation method is improved obviously on running efficiency and veracity of the results.
出处
《计算机工程与设计》
CSCD
北大核心
2008年第12期3208-3211,3265,共5页
Computer Engineering and Design
基金
国家863高技术研究发展计划基金项目(2004AA1Z2520)
军队网络互联与信息安全策略研究基金项目(2006QB1069)
关键词
中文分词
哈希结构
逆向最大匹配算法
分词词典
消除歧义
Chinese segmentation
hash structure
reverse directional maximum match method
dictionary mechanism
dispel ambiguity