基于免疫克隆特征选择和欠采样集成的垃圾网页检测

Web spam detection based on immune clonal feature selection and under-sampling ensemble

为解决垃圾网页检测过程中的"维数灾难"和不平衡分类问题,提出一种基于免疫克隆特征选择和欠采样(US)集成的二元分类器算法。首先,使用欠采样技术将训练样本集大类抽样成多个与小类样本数相近的样本集,再将其分别与小类样本合并构成多个平衡的子训练样本集;然后,设计一种免疫克隆算法遴选出多个最优的特征子集;基于最优特征子集对平衡的子样本集进行投影操作,生成平衡数据集的多个视图;最后,用随机森林(RF)分类器对测试样本进行分类,采用简单投票法确定测试样本的最终类别。在WEBSPAM UK-2006数据集上的实验结果表明,该集成分类器算法应用于垃圾网页检测:与随机森林算法及其Bagging和Ada Boost集成分类器算法相比,准确率、F1测度、AUC等指标均提高11%以上;与其他最优的研究结果相比,该集成分类器算法在F1测度上提高2%,在AUC上达到最优。

To solve the problem of ＂curse of dimensionality＂ and imbalance classification, a binary classifier algorithm based on immune clonal feature selection and Under-Sampling（ US） ensemble was proposed to detect Web spam. Firstly,major samples in training dataset were sampled into several sample subsets, which were combined with minor samples to generate several balanced training sample subsets. Then an immune clonal algorithm was proposed to select several optimal feature subsets. The balanced training subsets were projected to multiple views based on the optimal feature subsets. Finally,several Random Forest（ RF） classifiers were trained by these views of the training sample subsets to classify the testing samples. The testing samples＇ classifications were determined by voting. The experimental results on the WEBSPAM UK-2006 dataset show that the ensemble classifier algorithm outperforms these algorithms like RF, Bagging with RF and Ada Boost with RF, and its accuracy, F1-Measure, AUC（ Area Under ROC Curve） are increased by more than 11% respectively. Compared with several state-of-the-art baseline classification models, the F1-Measure is increased by 2% and the AUC reaches the optimum result using the ensemble classifier.