基于索引项权重的文本特征选择方法

Feature selection based on term weight for text categorization

为改善文本分类的效率和效果,降低计算复杂度,在分析了经典的特征选择方法后,提出加权的文本特征选择方法。该方法不仅利用数据集中文本的个数,还充分考虑到索引项的权重信息,并构造新的评估函数,改进了信息增益、期望交叉熵以及文本证据权。利用KNN分类器在Reuters-21578标准数据集上进行训练和测试。实验结果表明,该方法能够选出有效特征,提高文本分类的性能。

To improve the efficiency and effectiveness and reduce computational complexity for text categorization, text feature selection with term weight is prop6sed based on the classical method. This method not only used the numbers of documents in datasets, but also fully took the information of term weight into account in the text. Thus, new evaluation function is constructed. It works better than information gain, expected cross entropy and weight of evidence for text. Using K-Nearest neighbor classifier, Reuters-21578 is used as standard data collection. Experimental results show that the new method select good features and effectively improve the performance of text categorization.