分类视角下认知诊断测验项目区分度指标及应用

An Item Discrimination Index and Its Application in Cognitive Diagnostic Assessment on a ClassificationOriented View

在认知诊断中还没有指标能在无作答数据情况下直接评价项目的属性分类准确率或属性判准率。项目水平上的属性分类准确率,与项目属性向量、项目参数、先验分布和作答反应等有关。综合各个影响因素定义了项目水平上的属性期望分类准确率指标,并将其用于组卷。模拟研究显示:新指标可十分准确地评价项目的属性判准率,新指标对于项目筛选十分重要;以模式分类准确率为评价指标,基于新指标的组卷方法与经典的组卷方法表现相当。

The existing studies suggested that item quality is closely relevant to the number of attributes required by an item, item parameters, and the prior distribution of attribute patterns in cognitive diagnostic assessment. Several studies focused on the design of Q-matrix and showed that items required only one attribute are important for classification. There are some works that provided two basic sets of item discrimination index to measure discriminatory power of an item. The first one is based on descriptive measures from classical test theory, such as the global item discrimination index, and the second index is based on information measures from item response theory, including cognitive diagnosis index （CDI）, attribute discrimination index （ADI）, modified CDI and ADI. Results showed a strong relationship between these indices and the average correct classification rates of attributes. But their relationship to the indices may change as a function of the distribution of attributes. There lacks an item quality index as a measure of item＇s correct classification rates of attributes. The purpose of this study was to propose an item discrimination index as a measure of correct classification rate of attributes based on Q-matrix, item parameters, and the distribution of attributes. Firstly, an attribute-specific item discrimination index, called item expected attribute matched rate （EAMR）, was introduced. Secondly, a heuristic method was presented using EAMR for test construction. The first simulation study was conducted to evaluate the performance of EAMR under the deterministic input noisy ＂and＂ gate （DINA） model. Several factors were manipulated for five independent attributes in this study. Four levels of correlation between latent attributes, p=.00, p=.50, p=.75, and p=.95, were considered. Items were categorized into five groups according to the number of attributes measured by each item. Item discrimination power was set at three levels, high, medium, and low. High level meant relatively smaller guessing and slip parameters, which were randomly generated from a uniform distribution U（.05,.25）. Medium-level and low-level item parameters were randomly drawn from uniform distributions U（.05, .40） and U（.25, .45）. Next, 1000 items were simulated with the q-vector randomly selected from all possible attribute patterns measuring at least one attribute. Results showed that the new index performed well in that their values matched closely with the simulated correct classification rates of attributes across different simulation conditions. The second simulation study was conducted to examine the effectiveness of the heuristic method for test construction. The test length was fixed to 50 and simulation conditions are similar to those used in the first study. Results showed that the heuristic method based on the sum of EAMRs yielded comparable performance to the famous CDI. These indices can provide test developers with a useful tool to evaluate the quality of the diagnostic items. The attribute-specific item discrimination index will provide researchers and practitioners a way to select the most appropriate item and test that they want to measure with greater accuracy. It will be valuable to explore the applications and advantages of using the EAMR for developing item selection algorithm or termination rule in cognitive diagnostic computerized adaptive testing.