计算机自适应测验中试题泄露的实时监控方法研究与应用

The Study and Application of Real-Time Monitoring Methods for Item Leakage in Computerized Adaptive Tests

在连续施测下计算机自适应测验(CAT)中的试题被曝光的可能性急剧增加,因此需要对试题进行实时监控,当试题的参数发生显著性变化时必须将其进行强制“退休”.序贯监测程序(SMP)通过检测CAT中的试题统计特征的变化来判断试题是否泄露;然而在用SMP监控试题时会出现较大的Ⅰ类错误率,并且在一些条件下其统计检验力较低.该文以残差的R指标作为考生拟合统计量(PFS),与SMP方法相结合,构建了一种新的监测方法(PFS_SMP);该方法以被试作答信息为依据判断被SMP标记的试题是否泄露,从试题和被试这2个层面保证测验的安全性和公平性.最后,通过模拟实验和实证分析来对基于R的PFS_SMP的表现进行评价,实验结果表明:PFS_SMP方法能降低在SMP监测试题时的Ⅰ类错误,并能提高其统计检验力.

Computerized Adaptive Test(CAT)makes the possibility of each item being exposed increase,so the item needs to be monitored in real time.When the item parameters change significantly,it must be forced to"retire".The sequential monitoring program(SMP)is proposed in 2014 to determine whether an item is leaking by detecting changes in the statistical characteristics of the item in CAT.However,when using SMP to monitor the item,there will be a relatively high error rate of type I,and the statistical test will also have a greater impact.In this paper,based on the residual person fit statistic R,combined with the SMP method,a new monitoring method(PFS_SMP)is proposed.The PFS_SMP method can be applied to determine whether each respondent takes aberrant response behavior,and each item is known by the future respondents during the CAT,and to ensure the safety and fairness of the test.Finally,a simulation study and an empirical study are considered,and the results show that the PFS_SMP method can yield a well-controlled error rate of type I,and have a promising power as well.