分类不平衡对软件缺陷预测模型性能的影响研究

The Impact Study of Class Imbalance on the Performance of Software Defect Prediction Models

分类不平衡是指不同类别间样本数量分布不均衡的现象.在软件缺陷预测中,传统预测模型的性能可能会因数据集分类不平衡而受到影响.为了探究分类不平衡对软件缺陷预测模型性能的影响程度,该文提出一种分类不平衡影响分析方法.首先,设计一种新数据集构造算法,将原不平衡数据集转化为一组不平衡率依次递增的新数据集.然后,选取不同的分类模型作为缺陷预测模型,分别对构造的新数据集进行预测,并采用AUC指标来度量不同预测模型的分类性能.最后,采用变异系数C·V来评价各个预测模型在分类不平衡时的性能稳定程度.在8种典型的预测模型上进行实验验证,结果表明C4.5、RIPPER和SMO这3种预测模型的性能随着不平衡率的增大而下降,而代价敏感学习和集成学习能够有效提高它们在分类不平衡时的性能和性能稳定程度.与上述3种模型相比,逻辑回归、朴素贝叶斯和随机森林等模型的性能更加稳定.

Class imbalance refers to that the number of samples in different classes is unbalanced.In the process of software defect prediction,the performance of traditional prediction models may be affected by the class imbalance problem of datasets.In order to explore the impact of class imbalance on the performance of software defect prediction models,this paper presents an approach to analyzing the impact of class imbalance.First,an algorithm is designed to construct new datasets,which could convert an original imbalanced dataset into a set of new datasets with imbalance ratio increased one by one.Second,different classification models are selected as the defect prediction models to predict on these new constructed datasets respectively.Moreover,AUC metric is used to measure the classification performance of different prediction models.Finally,Coefficient of Variation(C·V)is applied to evaluate the performance stability of each prediction model with class imbalance.The empirical study is conducted on eight typical prediction models.The results show that the performance of three prediction models,C4.5,RIPPER and SMO,are decreased with the increasing of imbalance ratio.However,cost-sensitive learning and ensemble learning could improve their performance and performance stability with class imbalance.Compared with the above three models,the performance of Logistic Regression,Naive Bayes and Random Forest models are more stable.