允许CAT题目检查的区块题目袋方法

The Block Item Pocket Method to Allow Item Review in CAT

允许题目检查能够促进计算机化自适应测验(CAT)在实际中的应用。在不影响能力估计精度和测验公平性的前提下,允许CAT题目检查能够缓解考生考试焦虑,减少无关因素引起的测量误差。区块题目袋方法是连续区块方法与题目袋方法的结合,不仅能允许CAT题目检查,还能够弥补题目袋方法的不足。研究结果表明:(1)合理作答策略下,区块题目袋方法的估计精度在低能力水平上要优于题目袋方法;(2)在应对类似Wainer作答策略时,区块题目袋方法的估计精度在所有能力水平上均优于题目袋方法。(3)随着区块数的增加,区块题目袋方法的能力估计精度越接近无修改的基线水平。

Most computerized adaptive testing （CAT） do not allow examinees to review items because it will drastically decrease measurement precision and bring about extra cheating strategies （Wainer, 1993; Wise, 1996） Allowing item review is essential to make CAT comparable with traditional tests. It also matters in application. Item review enables examinees to correct mistakes due to carelessness, which can further improve the precision of ability estimation. No such option may cause some negative consequences for their overall performance especially in high-stake examinations, such as tension or anxiety （Vispoel, Henderickson, ＆ Bleiler, 2000）. Therefore, it is worth trying if allowing item review could alleviate problems mentioned at the beginning （Wise, 1996; Vispoel, 2000, 2005）. Several methods have been proposed, including the successive block method （Stocking, 1997） and the item pocket （IP） method （Han, 2013）. However, both methods are limited in some ways. Stocking＇s method does not allow examinees to skip items and requires a large number of blocks which may bring about some extra adverse effects because of frequent decision to go to next block. Han＇s method can avoid limitations of Stocking＇s. But it requires an appropriate IP size and may result in high bias in large IP size situation. The present study proposed the block item pocket （BIP） method which sets fewer but larger blocks with a proper total IP size. This method keeps advantages of Stocking＇s and Han＇s and overcomes their disadvantages. Two simulation studies of two response strategies were conducted to evaluate validity of the BIP method. Item parameters were randomly drawn from uniform distribution （b - U （-3, 3）） and （a - U （0, 2））. Each examinee was administered a fixed-length CAT with 30 items. The initial item for each examinee was randomly drawn from 0 - U （-0.5, 0.5）. For the CAT administration, the Maximum Fisher Information method was adopted to select items. The interim and final scores were estimated using MLE method in most conditions. When responses were less than 5 or when all answers were correct or wrong, EAP method was adopted. Each study contained five conditions： non-review, 1 blocks IP method, 2 blocks, 3 blocks and 6 blocks BIP method. Statistics like BIAS, MAE, and RMSE were used as evaluation criteria. Results indicated that： （1） BIP method had better estimate precision than IP method at low ability level under normal strategy; （2） When dealing with Wainer-like strategy, BIP method was far more precise than item pocket method at all ability levels; （3） As the number of blocks increased, estimate precision got closer to non-review condition. Advantages of this new method and future directions were discussed.