大标记集汉语字(词)Markov 语言模型的建立

Construction and Application of Large Symbol Set of Chinese Character/Word Markov Language Model

给出了一种基于快速排序和归并排序的高阶汉语大标记集Ｍａｒｋｏｖ统计语言模型的统计算法，并对算法的时间复杂性和空间复杂性进行了分析。依据这种统计算法，设计实现了一个汉语字（词）概率统计系统。通过对上千万字的汉语语料的统计，建立起了汉语字（词）一元、二元和三元Ｍａｒｋｏｖ模型，并对统计结果进行了分析。

This paper puts forward an algorithm which combines quick sorting and merge sorting to construct a large symbol set of Chinese character/word Markov Language models The time and the space complexity are discussed According to the algorithm,a Chinese character/word probability distribution computing system is introduced The unigram,bigram and trigram Chinese larguage models based on more than twenty million Chinese characters,and the results are analyzed From the experimental results we find statistical language models have a good performance in approaching the near constraint relationship of the Chinese language