摘要
分析了中文分词词典的机制,提出了一种改进的整词分词字典结构,并针对机械分词算法的特点,将其与概率算法相结合,探讨了一种中文自动分词概率算法。采用哈希及二分法对词典进行分词匹配。实验表明,该算法具有较高的分词效率和准确率,对于消去歧义词也有较好的性能。
Chinese segmentation mechanism is analyzed.An improved structure of segmentation dictionary is presented,and in view of the characteristics of the mechanical Chinese word segmentation,combined with probabilistic algorithm,a Chinese Word Automatic Segmentation probabilistic algorithm is discussed.Hashing and binary search is used to segmentation match.Experiment indicates that the algorithm can greatly improve the speed of Chinese segmentation and precision,and strengthen the processing of dispelling ambiguity.
出处
《计算机工程与应用》
CSCD
北大核心
2010年第3期125-127,130,共4页
Computer Engineering and Applications
基金
发展基金资助项目(WEB信息智能获取算法的研究
西南大学计算机与信息学院)
关键词
自动分词
分词算法
字典
歧义切分
automatic segmentation
segmentation algorithm
dictionary
ambiguity segmentation
