混合CHI和MI的改进文本特征选择方法

An Improved Text Feature Selection Method Mixed CHI and MI

特征选择在文本分类技术中是一个重要的关键部分,特征选择的好坏直接决定最终的分类结果。卡方统计量(CHI-square statistic,CHI)和互信息(mutual information,MI)是特征选择模块经常使用的方法。针对卡方统计量方法不考虑词频的不足,引入词频因子。考虑互信息方法倾向选择低频词的缺点,并研究特征词在不同类别文档内的分布情况对互信息方法的特征选择的影响,通过引入特征的词频因子和调节参数对卡方统计量方法和互信息方法进行改进,并混合改进后的卡方统计量方法和互信息方法,提出一种混合的特征选择算法(CHMI)。通过对CHI方法、MI方法、改进的CHI方法、改进的MI方法和CHMI方法进行实验对比,使用CHMI方法进行特征选择,使最终分类结果的查准率和F1值都有了提高,验证了CHMI方法的有效性。

Feature selection is a key part of text classification,and the choices of features directly determine the final classification result.CHI-square statistic(CHI)and mutual information(MI)are commonly used in feature selection.In order to solve the shortcoming of word frequency and analyze the distribution of the feature words in the documents within the categories,the word frequency factors and adjust parameter are introduced into the original CHI-square selection method and mutual information method,and then the improved CHI and improved MI are clustered to obtain a mixed feature selection algorithm(CHMI).By comparing CHI,MI,improved CHI,improved MI and CHMI with experiment,the CHMI improves the accuracy of classification and F1 values,which verifies its validity.