中文高频词串的抽取及其在语言模型中的应用

Chinese Frequent String Extraction and Application on Language Model

为准确抽取语料库中的高频词串,使其能更好地应用于语言模型中,提出了一种基于字串切分度的中文高频词串(CFS)抽取算法,并用该算法抽取出的CFS分别建立一元和二元语言模型.实验表明,基于CFS的语言模型能有效克服现有基于字和词的n元语法模型长距离相依性能较差的缺陷;同时,在模型困惑度、音字转换正确率上均优于已有基于净频次的CFS语言模型.

In order to extract the Chinese frequent strings（CFS） accurately and make better use in language models,a new method for CFS extraction using string segmentation degree is proposed.Unigram and bigram language models based on this CFS extraction method are built.Experiment shows that the CFS based language model can deal with the lack of long distance dependency problem in character and word based language model.It also shows that the CFS based language model has lower model perplexity and higher pinyin-to- character conversion correctness compared with the model based on previous CFS extraction method.