基于统计极差和变异系数的特征抽取研究

Research on Feature Extraction Based on Statistical Range and Coefficient of Variation

经典的文本频数DF从全局角度统计某个单词特征出现的文本数,而忽略了局部信息。文章针对传统DF文本特征抽取方法存在的不足,对DF方法进行了改进和优化,通过考虑单词特征的局部信息和全局信息,利用类别信息从局部统计单词在各个类上的分布,并结合分散性和变异性利用极差、变异系数和变形KL散度三种方式从全局角度度量单词重要性,提出了一种基于统计极差和变异系数的文本特征抽取方法。在英文文本数据集路透社Reuters-21578财经新闻和20Newsgroups新闻组以及中文文本数据集搜狗新闻语料库中进行了实验和测试,结果表明,该方法能够提升文本特征抽取的效果,与其他方法相比在多类别平衡文本数据集上更能显示其优越性。

The classical document frequency(DF) counts the number of documents with a certain word feature from a global perspective, while ignoring the local information. Aiming at the shortcomings of traditional DF text feature extraction methods, this paper optimizes the DF method. By considering the local and global information of word features, using the category information to locally count the distribution of words in each category, and combining the dispersion and variability to measure the importance of words globally by means of range, coefficient of variation and deformed KL divergence, the paper proposes a text feature extraction method based on statistical range and coefficient of variation. Experiments and tests are carried out in the English text dataset Reuters-21578 financial news and 20 newsgroups and the Chinese text dataset Sogou news corpus. The results indicate that the proposed method improves the performance of text feature extraction, and shows its superiority on multi-category balanced text data set compared with other methods.