以《汉学引得丛刊》为领域词表的先秦典籍自动分词探讨

Exploring of Word Segmentation for Fore-Qin Literature Based on the Domain Glossary of Sinological Index Series

[目的/意义]在人文计算兴起这一背景下,为了更加深入和精准地从古代典籍中挖掘出相应的知识,针对先秦文献进行自动分词的探究。[方法/过程]基于《汉学引得丛刊》中的《春秋经传注疏引书引得》制定词汇表,在由《春秋左氏传》和《晏子春秋》所构成的训练和测试语料上,通过条件随机场模型,结合使用统计和人工内省方法确定的特征模板,完成对先秦典籍进行自动分词的探究。[结果/结论]在先秦典籍自动分词的整个流程基础上,得到简单特征模板、内部特征模板和组合特征模板下的自动分词模型,最好的分词模型调和平均值达到97.47%,具有较强的推广和应用价值。在构建自动分词模型的过程中,通过融入内部和外部的特征知识,模型的精确率和召回率得到有效的提升。

[ Purpose/significance ] With the rising of humanities computing, in order to more deeply and accurately mine the corresponding knowledge from the ancient classics, the Fore-Qin Literature is automatically segmented in this pa- per. [ Method/processj Based on domain glossary of Zuo Commentary from the Sinological Index Series, the paper fini- shes the segmentation of Fore-Qin Literature on the corpus of train and test which consist of Zuo Commentary and Yanzi＇ s Spring and Autum Annals by the conditional random fields which uses the feature template determined by the method of statistics and rules. [ Result/conclusion ] The segmentation models based on simple feature template, internal feature template and combined feature template are obtained under the framework of word segmentation for Fore-Qin Literature. The best F-measure of segmentation model reaches 97.47% , which has a great potential for popularization and applica- tion. In the processof constructing the model, the precision rate and recall rate of segmentation model are effectively en- hanced by merging internal and external feature knowledge.