摘要
为提高整词二分词典机制的分词效率,分析现有分词词典机制,提出一种基于全Hash的整词二分词典机制。该机制将首字相同的词条按字数分组,并进行全词Hash,对Hash值相同的词条进行二分查找,从而减少词条匹配的次数。理论分析和实验结果表明,该机制的分词效率较高。
According to the low efficiency of the traditional binary-seek-by-word dictionary mechanism for word segmentation,this paper gives a binary-seek-by-word dictionary mechanism for word segmentation based on all-Hash by analyzing many old dictionary mechanisms.The new mechanism divides the dictionary entry into some groups by character number the entry has,it uses the Hash value of word to reduce the number of string finding.Theoretical analysis and experiment results show that the new mechanism improves the efficiency of word segmentation.
出处
《计算机工程》
CAS
CSCD
北大核心
2011年第21期40-42,共3页
Computer Engineering
基金
南京工程学院科研基金资助项目"基于Lucene的全文搜索引擎研究"(QKJB2009026)
关键词
中文分词
HASH函数
整词二分
逐字二分
最大匹配
Chinese segmentation
Hash function
binary-seek-by-word
verbatim binary search
maximum match