Feature selection is an important approach to dimensionality reduction in the field of text classification. Because of the difficulty in handling the problem that the selected features always contain redundant informa...Feature selection is an important approach to dimensionality reduction in the field of text classification. Because of the difficulty in handling the problem that the selected features always contain redundant information, we propose a new simple feature selection method, which can effectively filter the redundant features. First, to calculate the relationship between two words, the definitions of word frequency based relevance and correlative redundancy are introduced. Furthermore, an optimal feature selection（OFS） method is chosen to obtain a feature subset FS1. Finally, to improve the execution speed, the redundant features in FS1 are filtered by combining a predetermined threshold, and the filtered features are memorized in the linked lists. Experiments are carried out on three datasets（Web KB, 20-Newsgroups, and Reuters-21578） where in support vector machines and na？ve Bayes are used. The results show that the classification accuracy of the proposed method is generally higher than that of typical traditional methods（information gain, improved Gini index, and improved comprehensively measured feature selection） and the OFS methods. Moreover, the proposed method runs faster than typical mutual information-based methods（improved and normalized mutual information-based feature selections, and multilabel feature selection based on maximum dependency and minimum redundancy） while simultaneously ensuring classification accuracy. Statistical results validate the effectiveness of the proposed method in handling redundant information in text classification.展开更多
基金Project supported by the Joint Funds of the National Natural Science Foundation of China(No.U1509214)the Beijing Natural Science Foundation,China(No.4174105)+1 种基金the Key Projects of National Bureau of Statistics of China(No.2017LZ05)the Discipline Construction Foundation of the Central University of Finance and Economics,China(No.2016XX02)
文摘Feature selection is an important approach to dimensionality reduction in the field of text classification. Because of the difficulty in handling the problem that the selected features always contain redundant information, we propose a new simple feature selection method, which can effectively filter the redundant features. First, to calculate the relationship between two words, the definitions of word frequency based relevance and correlative redundancy are introduced. Furthermore, an optimal feature selection（OFS） method is chosen to obtain a feature subset FS1. Finally, to improve the execution speed, the redundant features in FS1 are filtered by combining a predetermined threshold, and the filtered features are memorized in the linked lists. Experiments are carried out on three datasets（Web KB, 20-Newsgroups, and Reuters-21578） where in support vector machines and na？ve Bayes are used. The results show that the classification accuracy of the proposed method is generally higher than that of typical traditional methods（information gain, improved Gini index, and improved comprehensively measured feature selection） and the OFS methods. Moreover, the proposed method runs faster than typical mutual information-based methods（improved and normalized mutual information-based feature selections, and multilabel feature selection based on maximum dependency and minimum redundancy） while simultaneously ensuring classification accuracy. Statistical results validate the effectiveness of the proposed method in handling redundant information in text classification.
