中文信息检索中词典机制分词算法的研究

Study of Segmentation Algorithm of Dictionary Mechanism Orienting Chinese Information Retrieval

中文自动分词是实现搜索引擎信息检索的基础,分词词典是汉语自动分词系统的一个重要组成部分,词典的加载和查询速度直接影响到分词系统的速度。文中在研究传统词典机制的基础上,分析了基于双字哈希词典机制对词条除首次字外剩余词的不足,给出了一种改进的双字哈希的词典机制。最后,文中对改进算法从准确率、分全率和分词速度等方面进行了测试,结果表明,改进后的分词算法在不提升已有典型词典机制维护复杂度的情况下,提高了词条匹配的查询速度和效率。

Chinese automatic segmentation is the base of the information retrieval search engine. Word dictionary is an important part of Chinese word segmentation system. The loading and querying efficiency is a key impact fact of the word segmentation system. Based on the study of the traditional dictionary mechanism, analyze the weak point of the double word hash dictionary, and propose a modified double hash dictionary. At last test the method from the accurate, full-rate, word speed, etc. With the result of the test, this improved hash mechanism enhances the entry speed and efficiency of matching queries, without completing the maintenance complexity of the traditional dictionary.