哪个测验Q矩阵更合理:基于DINA模型测验Q矩阵合理性侦查指标及其比较与应用

The development of the indexes to identify the rationality of test Q matrices based on DINA model

本研究对多个测验Q矩阵的相对合理性的比较与选用开展研究,采用Monte Carlo模拟与实证研究相结合的范式,探讨R_square、HCI、-2LL、AIC、BIC、residual、ABS_residual及本研究新开发的BIC2等八项指标在测验Q矩阵合理性侦查效果及其比较。研究发现:八项指标中,除BIC和BIC2两项指标的对测验Q矩阵相对合理性的平均正确识别率在95%以上,其余指标的平均正确识别率不足90%,整体而言,考虑样本容量及参数个数双重加权的BIC和BIC2两项指标的表现总体上优于其它几项指标;各项指标在不同Q矩阵错误类型下其正确识别率也不尽相同。

The purpose of cognitive diagnosis （CD） is to detect the internal psychological rules and the human processing mechanism. Since a cognitive diagnosis assessment would report the features about individuals, such as his/her current cognitive status, the cognitive procedure, and the processing skills and knowledge structures etc., it would help teachers and students learn a lot about an individual, and choose a more effective way to teach or learn. In short, CD would play an important role on an individual＇s development. More and more attention should be paid to the cognitive diagnosis assessment. With more and more applications of CD in practice, researchers and practitioners found out that the task to identify a test Q matrix was very hard. Even if a test Q matrix was established, it would also be difficult to evaluate its correctness. Virtually, there might be several possible test Q matrices provided by the experts at the same time. Practitioners are always confused about which might be the right. Corresponding to this phenomenon, they need to find a method to determine which one is the most appropriate one. This study attempted to propose some indexes which might be used for the selection of a test Q matrix. In order to verify the effectiveness of the indexes, a comparison study was done. It aimed to find out the advantages and disadvantages of each index, and to get an idea of which indexes might be used to choose Q matrices. All these kinds of information could be very useful for the applicants. In this study, some test Q matrices with different degrees of correctness were provided, and the DINA model was applied. Both the Monte Carlo simulation method and the real data analysis were used; all the results were indicated as follows： Firstly, according to these eight indexes, the average correct identification rate （ACIR） of the BIC and BIC2 was greater than 95 percent. The ACIRs of the other indexes were less than 90 percent; some indexes were even less than 50 percent. Secondly, the ACIRs varied according to different categories of incorrect Q matrices. Considered with the E error type, the ACIRs of these indexes were relatively low, most of them were less than 50 percent, except the BIC2 index. Thirdly, weighted by both the size of samples and the number of parameters, the performances of BIC and BIC2 were better than those of the rest. From the above results, it is showed that the BIC and BIC2 might be the best indexes for the selection of test Q matrix when multiple test Q matrices were proposed.