基于关联规则的自动构词算法研究

Research and Application on Auto-word Building

词语是中文文本的基本元素,汉语语言模型在中文文本挖掘中起关键作用。中文文本挖掘是高维度的数据处理技术,挖掘算法对维度的大小比较敏感,因此挖掘效果依赖于词库的质量。另外,现存的汉语语言模型一般都是基于统计的,比如N-gram语言模型以及各种改进模型都具有较高的计算复杂度。为降低语言模型的计算复杂度、提高词库的质量和构词效率,借鉴关联规则理论对中文词语进行定义,在此基础上构建Auto-word自动构词算法。该算法可以从大量中文语料库中动态地构造词表,并以此为基础进行中文文本挖掘工作。最后通过实验证明了提出的自动构词算法的有效性。

Words are the basic elements of Chinese text,and Chinese language model plays a key role in Chinese text mining.Text classification is a data mining technology with high dimensions and most of the classifying algorithms are sensitive to the dimensions.As a result,the classification depends on the quantity of vocabularies.Besides,most of current Chinese language models are based on statistical theory,such as N-gram model and other improved models.However,these statistical models are disadvantaged with computational complexity.In order to improve the quantity and efficiency,this paper gave Chinese words a new definition based on association rules,and proposed the Auto-word algorithm,by which a word vocabulary is constructed automatically and used for Chinese text mining.Finally,the efficiency of the Auto-word algorithm was proved by experiment.