先秦文献《孟子》自动分词方法研究

Methodological Study of Automatic Word Segmentation in Pre-Qin Document Mencius

自动分词是中文信息处理重要的基础课题。文章主要探讨了先秦文献《孟子》基于条件随机场统计模型的自动分词方法和利用相关注疏文献的自动分词方法等自动分词技术。自动分词实验结果表明,这两种分词方法效果显著,词语F值和小句F值均达到了较高的水平。在用不同的方法对《孟子》进行自动分词时,在词语F值的统计指标外,首次尝试引入了小句F值这一统计指标。

The automatic word segmentation is a basic subject in Chinese language information processing. This paper discusses the automatic word segmentation technology applied in the pre - Qin text Mencius. It is based on condition random field statistic model and utilizes the relevant ancient annotations and comments. The effect of these two methods is significant. The F values of both words and short sentences have reached higher levels. When using different method in automatic segmentation on Mencius, we are the first to introduce the statistic indicator of the F value of short sentences besides that of words.