基于互信息的统计语言模型平滑技术

A Smoothing Technique for Statistical Language Model Based on Mutual Information

数据平滑主要是用来解决统计语言模型在实际应用中数据稀疏问题。现有平滑技术虽然已有效地对数据稀疏问题进行了处理,但对已出现事件频率分布的合理性并没有作出有效的分析。本文则针对二元模型,提出了一种基于互信息的平滑技术,其基本思想是根据模型中每个二元对的互信息的高低对其概率进行折扣或补偿,并用极小化困惑度原则体现了模型的合理性。实验结果表明该技术优于目前常用的Katz平滑技术。

Smoothing techniques are mainly used to solve the problem of sparse data for statistical language model. The present smoothing techniques have solved the data sparse problem effectively but have not further analyzed the reasonableness for the frequency distribution of events occurring. This paper presents a new kind of smoothing technique based on the mutual information for Bi-gram model. The model parameters, probabilities for bigram, are discounted or compensated according to the mutual information, whose rationality is indicated by minimizing the perplexity. The experimental results show that this technique outperforms the commonly used Katz smoothing technique.