垃圾邮件处理中LDA特征选择方法

LDA based feature selection for spam filter

垃圾邮件处理是一项长期研究课题,越来越多的文本分类技术被移植到垃圾邮件处理应用当中。LDA(Latent Dirichlet Allocation)等topic模型在自动摘要、信息获取和其他离散数据应用中受到越来越多的关注。将LDA模型作为一种特征选择方法,引入垃圾邮件处理应用中。将LDA特征选择方法与质心+KNN分类器结合,得到简单的测试用垃圾邮件过滤器。初步实验结果表明,基于LDA的特征选择方法优于通常的IG、MI特征选择方法;测试过滤器的过滤性能与其他过滤器相当。

Spam filtering is a long-drawn research issue.More and more text categorization techniques are replanted for spam filtering.Latent Dirichlet Allocation（LDA） and other related topic models are increasingly popular tools for summarization,manifold discovery and other application in discrete data.The LDA is introduced into spam filtering as a feature selection tool.Combined the LDA with a simple centroid-based ＋ kNN classifier,a test spam filter is got.The experiment result shows that the features selected by LDA outperform the baseline features selected by IG and MI, and the test filter is comparative to other filters.