基于机器学习的无需人工编制词典的切词系统

A MACHINE LEARNING BASED WORD SEGMENTATION SYSTEM WITHOUT MANUAL DICTIONARY

本文介绍了一个基于自学习的无需人工编制词典的切词系统.首先用统计方法建立起附有相关度的切词词典.然后,提出将切词问题转化为一个有向图中求解最大加权路径问题,并提出利用词典中的相关度信息切分文本的一个新算法.最后,我们对词典和切词的质量都作了系统的分析,并与其他方法作了性能比较.

This paper describes a Chinese word segmentation system without the help of manual dictionary, which is based on automatic machine learning. Using traditional contingency table method, we first, create a dynamic dictionary with coincidence information. Then we argue that the problem of word segmentation can be transfered to the problem of path searching in weigthed graph, and provide a fast algorithm to segment texts in the corpus using the coincidence information. At last, we systematically analysis the accuracy and efficiency of this dictionary, and compare our method with others.