一种基于层叠CRF的古文断句与句读标记方法

Method of sentence segmentation and punctuating for ancient Chinese literatures based on cascaded CRF

针对利用自然语言理解技术进行古汉语断句及句读标注的主要挑战是数据稀疏问题,设计了一种六字位标记集,提出了一种基于层叠式CRF模型的古文断句与句读标记方法。基于六字位标集,低层模型用观察序列确定句子边界,高层模型同时使用观察序列和低层的句子边界信息进行句读标记。实验在5M混合古文语料上分别进行了封闭测试和开放测试,封闭测试断句与句读标注的F值分别达到96.48%和91.35%,开放测试断句与句读标注的F值分别达到71.42%和67.67%。

Data sparseness is a primary challenge in sentence segmentation and punctuating for ancient Chinese literatures using natural language processing technology. In order to overcome this difficulty, designed a 6-tag set and proposed a method based on cascaded conditional random fields. The main idea was as follows ： basing on the 6-tag set, a low level model deter- mined the boundaries of sentences according to observation sequence and a high level model punctuated sentences taking con- sideration of both observation sequence and low level＇ s results. Done close test and open test based on approximate 5M mixed corpus respectively. The F measure of sentence segmentation and punctuation were 96.48%. and 91.35% respectively in close test, and those were 71.42% and 67.67% respectively in open test.