基于受限语料库的语言平滑算法比较研究

Comparative Study on Algorithms of Limited Corpus Language Model

随着网络的广泛应用和科技的高速发展,人们所接收信息急剧增加,机器翻译面临强大的市场需求。从现存文本资料中提取语言模型,是整个机器翻译系统的重点,决定了翻译系统的性能表现。用于特定领域的文本翻译系统,往往受到相关文本缺少的困扰,无法通过大规模语料库的建设来训练语言模型,由此而产生了严重的数据稀疏问题。通过实验研究了受限语料库下语言模型平滑算法的选择。实验结论表明,在语料库极度受限的情况下,Good-Turing能够发挥其低频词汇重估优势,良好解决训练语料库的数据稀疏问题。通过该方法,可以提高在语料受限条件下语言模型的性能。

In recent years,with the rapid development of science and technology and the widespread application of Internet,information increases dramatically.Training language model from corpus plays an important role in improving system performance For specific areas translation task,it is often plagued by the lack of relevant texts,fail to construction of large-scale corpus to train the language model,resulting in serious data sparse problem.This paper focuses on choosing smoothing algorithms under limited corpus language model.Through several comparative experiments,it can be concluded that Good-Turing method can leverage its low-frequency lexical revaluation advantage,and solve the problem caused by data sparse efficiently,and also improve the efficiency of language model under limited corpus.