二元语法中文分词数据平滑算法性能研究

Performance of smoothing algorithm in Chinese word segmentation by bigram

将多种平滑算法应用于基于二元语法的中文分词,在1998年1月人民日报语料库的基础上,讨论了困惑度和实际分词性能之间的关系,对比分析各平滑算法的实际性能,结果表明,简单的加值平滑算法性能最优,封闭精度、召回率分别为99.68%、99.7%,开放精度、召回率为98.64%、98.74%。

This paper discusses the relationships between complexity and real performance based on the corpus of People＇s Daily of January ,1998 , compares the performance of multiple smoothing algorithms.The result reveals that additive smoothing is the best with 99.68% on precision,99.7% on recall in close test,and 98.64% on precision,98.74% on recall in open test.