古汉语句子切分与句读标记方法研究

Research on Sentence Segmentation and Punctuation in Ancient Chinese

利用自然语言理解技术进行古汉语断句及句读标注的主要挑战是数据稀疏问题.为了解决这一难题,设计了一种六字位标记集,提出了一种基于层叠式条件随机场模型的古文断句与句读标记方法.基于六字位标集,低层模型用观察序列确定句子边界,高层模型同时使用观察序列和低层的句子边界信息进行句读标记.实验在5 M混合古文语料上分别进行了封闭测试和开放测试,封闭测试断句与句读标注的F值分别达到96.48%和91.35%,开放测试断句与句读标注的F值分别达到71.42%和67.67%.

Data sparseness is a primary challenge in sentence segmentation and punctuation in ancient Chinese using natural language processing technology. In order to overcome this difficulty, a 6-tag set was designed and a method based on cascaded Conditional Random Fields was proposed. The main idea is as follows： based on the 6-tag set, a low level model determines the boundaries of sentences according to observation sequence and a high level model punctuates sentences taking consideration of both observation sequence and low level＇s results. Close test and open test were done based on approximate 5M mixed corpus respectively. The F measure of sentence segmentation and punctuation are 96.48% and 91.35% respectively in close test, and those are 71.42% and 67.67% respectively in open test.