结合局部歧义词网格与条件随机场的双层分词法

A Double-layer Word Segmentation Combined with Local Ambiguity Word Grid and CRF

本文提出了一种结合局部歧义词网格与条件随机场的双层中文分词模型。首先在底层使用局部歧义词网格对文本进行粗切分，并将切分结果作为一项特征提供给高层的条件随机场模型；然后使用条件随机场模型对文本进行标注分词。局部歧义词网格方法能够检测分词过程中产生的歧义问题，条件随机场模型能够平衡对待词表词和未登录词。两种方法的结合能够较好地解决分词中的分词歧义和未登录词问题。本文在国际ee文分词评测活动Bakeoff2005提供的PKU和MSRA语料上对该双层分词模型进行了系统封闭测试，并进行了四字位标注集与六字位标注集的对比。实验结果的最佳F值分别达到了95．1％和97．1％，优于单独使用条件随机场的分词效果。此外，开放测试的实验结果表明该模型也具有一定的实际意义。

This paper presents a double-layer model of Chinese word segmentation based on the combination of Local Ambiguity Word Grid and Conditional Random Fields. Firstly, the Local Ambiguity Word Grid algorithm is used to generate rough segmentation results in the lower level. Then, the text is segmented again based on CRF, where the rough results are set as one feature. The Local Ambiguity Word Grid algorithm has the advantage of detecting ambiguity from the process of Chinese word segmentation, while CRF can cope with vocabulary and out-of-vocabulary word equally. Therefore, the hybrid Local Ambiguity Word Grid and CRF approach is the effective resolution for the ambiguity and out-of-vocabulary word. The system is closed tested in the MSRA and PKU testing sets that are provided by the SIGHAN2005 Chinese Language Processing Bakeoff, along with the comparison between four characters and six characters in a set of label. The experiments show that F-measures of the MSRA and PKU testing sets in the closed test reach 97.1% and 95.1% respectively. Additional, the experimental results of open test reveal the practical application of the model.