基于CRF的先秦汉语分词标注一体化研究

CRF Based Research on a Unified Approach to Word Segmentation and POS Tagging for Pre-Qin Chinese

该文探索了古代汉语,特别是先秦文献的词切分及词性标注。首先对《左传》文本进行了词汇处理(分词和词性标注)和考察分析,然后采用条件随机场模型(CRF),进行自动分词、词性标注、分词标注一体化的对比实验。结果表明,一体化分词比单独分词的准确率和召回率均有明显提高,开放测试的F值达到了94.60%;一体化词性标注的F值达到了89.65%,比传统的先分词后标注的"两步走"方法有明显提高。该项研究可以服务于古代汉语词汇研究和语料库建设,以弥补人工标注的不足。

This paper explores the cross field between NLP and ancient Chinese, particularly the pre-Qin documents. The text of ＂Zuo Zhuan＂ is firstly analyzed after manual segmentationand POS tagging. Then the Conditional Ran- dom Fields model （CRF） is adopted for the word segmentation （WS）, POS tagging （PT） and a unified process of WS and PT, respectively. The precision and recall of the unified approach are much higher than the independent WS and PT in the open test, with a F-score of 94.60% in WS and 89.65% in PT. This method is suitable for the study of ancient Chinese vocabulary and corpus construction, and can be applied to compensatethe manual tagging.