概率估算方法对中文文本特征提取质量的影响

Effects of Probability Estimate on Quality of Extracting Feature in Chinese Text Categorization

在文本自动分类中,目前有词频和文档频率统计这两种概率估算方法,采用的估算方法恰当与否会直接影响特征抽取的质量与分类的准确度。本文采用K最近邻算法实现中文文本分类器,在中文平衡与非平衡两种训练语料下进行了训练与分类实验,实验数据表明使用非平衡语料语料时,可以采用基于词频的概率估算方法,使用平衡语料语料时,采用基于文档频率的概率估算方法,能够有效地提取高质量的文本特征,从而提高分类的准确度。

In text categorization,there are two kinds of probability estimation methods that they based on word frequency and document sfrequency,the stimation method＇ approiproteion or not will directly affect on the quality of feature extraction and classification accuracy.In this paper,there are train and classification experiment in the balanced and unbalanced China training data based on K-nearest neighbor algorithm as Chinese text classifier,the experimental data show that there are good effect on text feature extraction when word frequency is used in balanced data and document frequency is used in inbalanced data,which it can get high-quality texts,and improve the accuracy of classification.