层级一致性指标的多级评分拓展

Generalized Hierarchy Consistency Index for Cognitive Diagnosis Assessment

在认知诊断评估实践中,属性层级合理性的验证非常重要,而现有指标仅停留在0-1计分测验,无法适应考试形式和评分方式多样化的实践需求。研究将0-1计分层级一致性指标(MHCI)拓展至多级计分的层级一致性指标(GHCI),模拟和实证研究结果表明:(1)GHCI具有和MHCI相同的本质含义,考虑了父项目和子项目得分的多种可能性,从而将MHCI纳入GHCI体系;(2)在多级或混合计分情境,MHCI会有信息损失,容易发生低估,且易受转换比例的影响;(3)GHCI在模拟和实践情境均具较好的适宜性,拟合截断值的设置可依属性层级而定。

In educational practice of Cognitive Diagnostic Assessment （CDA）, it is crucial to validate the reasonability of the hierarchical structure of attributes, because it can affect the quality of CDA and the accuracy of classification of examinees directly. Several indices that validate the reasonability of the hierarchical structure of attributes have been developed by researchers. Based on Attribute Hierarchy Method （AHM）, Cui, Leighton, Gierl and Hunka （2006） developed Hierarchy Consistency Index （HCI） to detect the degree to which an observed examinee response pattern is consistent with the attribute hierarchy; also Ding, Man, Wang and Luo （2011） modified the HCI, then developed a new index- Modified HCI （MHCI）. In addition, Guo （2012） proposed the hierarchy misfit index （HMI） to detect misfitting item response vectors. Although these indices can be used to detect the misfits, all of these indices are suitable for dichotomous items. For polytomous items, in order to validate the reasonability of Attribute Hierarchy （AH）, researchers, in general, transform these polytomous items into dichotomous items according to some prespecified rules, then calculate the HCI （Kang, Wu, Chen, ＆ Zeng, 2015; Kang, Xin, ＆ Tian, 2013） or MHCI （Ding et al., 2012）. However, it will lose some details when transform polytomous items into dichotomous, therefore produce larger errors （Ding, Wang, ＆ Luo, 2014） and underestimate the consistence of AH. The purpose of this study is to extend the Modified HCI （MHCI） to a new HCI that is suitable for both dichotomous and polytomous items, and we name the new HCI as Generalized Hierarchy Consistency Index （GHCI）. To evaluate the suitability of GHCI, a simulation study and an empirical study are employed. For the simulation study, to compare the GHCI and MHCI under different conversion ratio, two independent variables are manipulated： type of AH and proportion of transformation. The AH has 4 levels （linear, convergent, divergent, and unstructured） and the proportion of transformation has 5 levels （GHCI, 60%MHCI, 2/3MHCI, 75%MHCI, 100%MHCI）. The control variables are the number of attributes, K = 5, and the number of examinees, N = 2000. Matlab R2013a software was applied to generate examinees＇ response matrix and compute the GHCI and MHCI. Results showed that： （1） Both GHCI and MHCI were affected by the AH type, and the linear AH had the best GHCI and MHCI, then it was convergent AH, after that was divergent AH and the unstructured AH was the worst. （2） GHCI had larger means than MHCI regardless of AH and proportion of transformation. The empirical study shows the same consistent pattern with the simulation study.