具有认知诊断功能的多阶段自适应测验及其影响因素研究

Affecting Factors of Multi-Stage Testing with Cognitive Diagnosis

MST结合了纸笔测验和CAT的优势,现阶段在美国的许多大型考试中得到了应用。本文对CD-MST的可行性进行研究。CD-MST具有认知诊断和自适应的功能;它计算速度较快,允许考生返回检查和修改,在测验的编制上更容易控制。本研究考察了不同的CD-MST在线组卷方法的表现。通过模拟研究发现:四种在线组卷方法中,PWKL的PCCR最低,MPWKL、GDI和SHE的PCCR不存在差异。PWKL受阶段数量的影响最大。随着阶段数量的增加,PWKL的结果越来越接近于其他三种方法。

Multi-stage Testing is a computerized testing which aims to improve the disadvantages of CAT, it incorporates most of the advantages from CAT and linear testing. Cognitive Diagnosis Assessment aims to determine whether or not examinees have each of many attributes or skills underlying responses to test items. In contrast to IRT, CDA provides a more detailed evaluation of the strengths and weaknesses of students. This study presents Cognitive Diagnostic Multi-stage Testing based on the thoughts ofMST, CDA, CD-CAT and OMST, in order to increase the application of CDA. CD-MST is a new idea which combines CD with MST, it has many advantages. CD-MST has the functions of cognitive diagnosis and adaptive, it can use less items and time to learn the examinee＇s knowledge c0ndition, it also can offer feedback to examinee regarding his or her strengths and weaknesses; compared with CD-CAT, it has speed superiority, and allows the examinees to go back to check and revise; it is also a flexible testing, the test takers can arrange the testing designs according to their demand. CD-MST can solve some practical problems, it has theoretical and practical values. Item selection strategy is the most important content of CD-CAT, because it can influence the test security, efficiency, the utilization of item bank and the accuracy directly. On-the-fly assemble methods of CD-MST have the same meaning and function. Zheng and Chang （2015）transferred item selection strategies of CAT to OMST. von Davier and Cheng （2014） suggested that item selection strategies of CD-CAT can be used in CD-MST. We use simulation study to verify how well it works when item selection strategy of CD-CAT transfers to on-the-fly assemble of CD-MST. Two simulation studies were used to illustrate on-the-fly assemble of CD-MST under compensation and non-compensation model. The non- compensation model was D1NA model, and the compensation model was rRUM model. On-the-fly assemble methods of CD-MST were PWKL, SHE, MPWKL and GDI. The number of examinee was 3000, item bank was 1240, number of attributes was 5. Item quality was a contributory factor, it has four levels, HD-LV, HD-HV, LD-LV, LD-HV respectively. The value index was the PCCR. The results showed that compared with MPWKL, GDI and SHE, the recovery rate of PWKL was poorer, especially under LD-LV. This is because PWKL relies on the estimate of knowledge state ajl. ajl is inaccurate in initial stage of testing, PWKL would be influenced. CD-MST has fewer adaptive frequencies, so PWKL is not fit CD-MST. MPWKL, GDI and SHE had the same recovery rate, and have good performance. The recovery rate of CD-MST was influenced by the quality of item bank. The result of HD-HV was the best, HD-LV took the second place, LD-HV was difference with the former significantly, and LD-LV was the last.