认知诊断缺失数据处理方法的比较:零替换、多重插补与极大似然估计法

Comparison of missing data handling methods in cognitive diagnosis:Zero replacement,multiple imputation and maximum likelihood estimation

数据缺失在测验中经常发生,认知诊断评估也不例外,数据缺失会导致诊断结果的偏差。首先,通过模拟研究在多种实验条件下比较了常用的缺失数据处理方法。结果表明:(1)缺失数据导致估计精确性下降,随着人数与题目数量减少、缺失率增大、题目质量降低,所有方法的PCCR均下降,Bias绝对值和RMSE均上升。(2)估计题目参数时,EM法表现最好,其次是MI,FIML和ZR法表现不稳定。(3)估计被试知识状态时,EM和FIML表现最好,MI和ZR表现不稳定。其次,在PISA2015实证数据中进一步探索了不同方法的表现。综合模拟和实证研究结果,推荐选用EM或FIML法进行缺失数据处理。

The problem of missing data is common in research,and there is no exception for cognitive diagnostic assessment(CDA).Some studies have revealed that both the presence of missing values and the selection of different missing data processing methods would affect the results of CDA.Therefore,it is necessary to attach more attention to the problem in CDA and choose appropriate methods to deal with it.Although the problem in CDA has been explored before,previous studies did not consider multiple imputation(MI)and full information maximum likelihood(FIML),which are widely used in the field of missing data analysis.Moreover,previous studies neglected the comparison using empirical data and saturation models such as GDINA model.In summary,the main purpose of this study are to introduce MI and FIML into CDA,thus making a comprehensive comparison of different missing data handling methods,and further putting forward suggestions for handling missing data in practice.Simulation study considered six factors:(1)Sample size:200 participants,400 participants,and 1000 participants;(2)Test length:15 test items and 30 test items;(3)Quality of items:high quality,medium quality,and low quality;(4)Missing data mechanisms:missing completely at random(MCAR),missing at random(MAR),and missing not at random(MNAR);(5)Missing rate:10%,20%,and 30%;(6)Missing data handling methods:zero replacement(ZR),MI-CART,MI-PMM,MI-LOGREG.BOOT,Expectation-Maximization algorithm(EM),and FIML.The GDINA model was used,and the analysis process was realized by the GDINA package in R software.Secondly,the PISA 2015 computer-based mathematics data were applied to compare the practical value of the proposed methods.The results of simulation study revealed that:(1)Missing data results in a decrease in estimation accuracy.The absolute value of Bias and RMSE both increased and PCCR values of all methods decreased as the sample size,test length and the quality of the items decreased and the missing rate increased;(2)When estimating item parameters,EM performed best,followed by MI.Meanwhile,FIML and ZR methods were unstable;(3)When estimating the KS of participants,EM and FIML performed best as the missing data mechanism was MAR or MCAR.When the missing data mechanism was MNAR,EM,FIML and ZR performed best.The empirical study results further supported the simulation research results.It showed that:(1)For all empirical indicators,EM,FIML,and MI-PMM perform best on one or more indicators;(2)The results obtained under the empirical study and simulation study under the MNAR mechanism are very similar;(3)EM performs well on all indicators,and ZR and FIML methods are slightly worse than EM,followed by MI-PMM,LOGREG.BOOT and MI-CART.In addition,based on the research results,the following suggestions were provided:(1)EM and FIML should be the first choice.However,if researchers do not want to get the complete data set,FIML could be used as a priority for missing data handling;(2)When the missing data mechanism was MAR or MCAR and the test length was not enough,researchers should avoid using the ZR method to deal with missing data.Finally,this paper ends with the prospects of future researches:(1)The multilevel scoring situation should also be studied;(2)The effectiveness of these methods should be tested in longitudinal research;(3)The performance of more methods of information matrix can be further compared in calculating the standard error to handle missing data;(4)Future research could focus on the missing mechanisms of data onto the real data.