基于字的词位标注汉语分词

Word-position-based tagging for Chinese word segmentation

近年来基于字的词位标注方法极大地提高了汉语分词的性能,该方法将汉语分词转化为字的词位标注问题,借助于优秀的序列标注模型,基于字的词位标注汉语分词方法逐渐成为汉语分词的主要技术路线。该方法中特征模板选择至关重要,采用四词位标注集,使用条件随机场模型进一步研究基于字的词位标注汉语分词技术,在第三届和第四届国际汉语分词评测Bakeoff语料上进行封闭测试,并对比了不同特征模板集对分词性能的影响。实验表明采用的特征模板集:TMPT-10′较传统的特征模板集分词性能更好。

The performance of Chinese word segmentation has been greatly improved by the word-position-based approaches in recent years.This approach treated Chinese word segmentation as a word-position tagging problem.With the help of a powerful sequence tagging model,the word-position-based method could quickly rose as a mainstream technique in this field.Feature template selection was crucial in this method.This technique was further studied via using four word-positions and conditional random fields.Closed evaluations were performed on corpus from the third and the fourth international Chinese word segmentation Bakeoff,and comparative experiments were performed on different feature templates.Experimental results showed that the feature template set：TMPT-10＇ was much better performance than the traditional template set.