一种面向文本分类的特征迁移方法

Feature Transfer Learning for Text Categorization

传统的文本分类方法假设训练集与测试集中的特征词服从相同的概率分布,但在实际应用中,以上假设存在偏差,会影响到最终的分类结果。针对这一情况,本文采用迁移学习,通过计算特征词的迁移量对训练集中向量空间模型进行修正,最终使训练集与测试集中特征词的分布概率趋于一致。将提出的方法应用于中文垃圾邮件过滤与中、英文网页分类中,在CHI统计特征选择基础上进行特征迁移,实验结果表明新方法可以有效消除特征词分布的差异性,使文本分类的各项指标明显提高。

Traditional text classification methods assume that feature words in the training set and test set follow the same probability distribution. Nevertheless, deviations exist in a practical application, which can affect the final classification results. To solve the problem, a feature transfer learning algorithm for text categorization is proposed. By calculating the transfer volume and amending the vector space model in the training set, the distribution probability of feature words can be reconciled for the training set and test set. Experiments on Chinese spam filtering and web page classification data sets demonstrate that the proposed method can eliminate the dissimilarity of distributions of feature words, and improve the va rious indexes of test classification evidently.