一种改进整词二分法的中文分词词典设计

Design of Chinese word segmentation dictionary based on traditional dictionary

汉语自动分词是汉语信息处理的前提,词典是汉语自动分词的基础,分词词典机制的优劣直接影响到中文分词的速度和效率。针对常用的分词方法——正向匹配分词法的特征,在已有整词二分词典的基础上加以改进,设计了一种基于Hash索引、二叉树以及动态一维数组结构的中文分词词典,相对于传统的基于Hash索引的分词词典,减小了词典的空间复杂度,提高了汉语分词的速度,最主要的是改进后的词典在增添新词时比以往的词典更为容易。

Chinese word segmentation dictionary is the important base of Chinese information processing system. The arithmetic of the dictionary influences the speed and efficiency of segmentation. This paper designs a new Chinese word segmentation dictionary based on the improved traditional binary-seek-by-word dictionary. The dictionary uses less memory. The experiment shows that the dictionary has high efficiency and is easy to build.