大规模汉语语料库中任意n的n-gram统计算法及知识获取方法

Algorithm of n gram Statistics for Arbitrary n and Knowledge Acquisition Based on Statistics

本文提出并实现了一种大规模汉语语料库中字、词级任意ｎ的ｎ－ｇｒａｍ统计算法，本算法可以一次性统计出所有不大于任意ｎ（本文ｎ取为２５６）的字、词级ｎ－ｇｒａｍ，可将传统ｎ－ｇｒａｍ统计时的指数空间开销变为线性的，且与所统计的元数无关。基于这种ｎ－ｇｒａｍ的统计，本文还进行了汉语信息熵的计算及字、词级知识获取的研究。

A new algorithm of n gram statistics for arbitrary n at word or phrase level is proposed and realized in this paper,with which the n gram for all n at word or phrase level can be calculated at the same time. Based on the n gram,the Chinese information entropy and knowledge acquisition at word or phrase level have also been studied.The algorithm and its result have been integrated with a MT system.