计算机辅助诊断模型内部验证方法的定量评价

Quantitative evaluation of internal validation methods for computer-aided diagnosis scheme

目的定量比较4种常用的内部验证方法,为评价计算机辅助诊断模型性能时选择验证方法提供参考依据。方法利用Logistic回归模型完成大样本集(n=415)和小样本集(n=76)下的胰腺癌诊断任务,分别采用保持法、k折交叉验证法、留一法和0.632 Bootstrap法共4种内部验证方法,并用诊断的正确率、敏感度、特异度和ROC曲线下面积评价诊断的稳定性、偏倚和运算效率。结果对大、小样本集,0.632 Bootstrap验证方法得到的正确率、敏感度、特异度和ROC曲线下面积的标准误分别为0.012、0.014、0.010、0.010以及0.013、0.014、0.010、0.011,均小于其他验证方法,其他方法均不同程度地高估或低估模型性能。结论考虑验证的简洁有效性,k折交叉验证法在大样本量的情况下即可达到内部验证的最佳效果,在小样本量情况下推荐使用0.632 Bootstrap进行验证。

Objective To quantitatively compare four commonly used methods in order to provide reference on the selection of internal validation methods for evaluating a computer-aided diagnosis model.Methods Logistic regression model was used for a diagnostic task on pancreatic cancer datasets with small and large sample sizes（ 76 and 415,respectively）. Four internal validation methods,hold-out,leave-one-out,kfold cross validation and 0. 632 Bootstrap,were used and compared. Diagnosis model stability,bias and efficiency were measured by accuracy,sensitivity,specificity and area under the ROC curve. Results 0. 632 Bootstrap validation method was with the minimum standard errors of accuracy,sensitivity,specificity and area under the ROC curve on both large-and small-size datasets,i. e. 0. 012,0. 014,0. 010,0. 010,and 0. 013,0. 014,0. 010,0. 011,respectively. Other methods underestimated or overestimated the model performance to certain degree. Conclusions Considering the simplicity and effectiveness of these validation methods,it is recommended that k-fold cross validation is preferable on the relative large-size dataset and 0. 632 Bootstrap method on the small one.