文件筐测验中评分者信度的误差分析

Error analysis of inter-rater reliability in in-basket test

目的探讨文件筐测验中评分者信度的影响因素。方法培训程度不同的2组评分者采用2种方法对代表2个复杂程度不同的能力试题评分，运用概化理论分析评分者信度差异。结果复杂能力用分析法较好（p、r、pr变异分量估计值为0．743，0．014，0．395）；用分析法评价复杂能力时，低培训组的评分（Ep25=0．8519，Ф5=0．8519）最接近高培训组的评分（E025=0．9039，Ф5：0．9010）。结论方法因素与评估内容的交互作用对评分者信度有影响。

Objective The research discusses the factors which to affect the inter-rater reliability in inbasket test. Methods Raters who belong to 2 different-trained-degree groups mark the same items using Degree Marking Method and Taking-out Marking Method. These items represent 2 different complexity degrees. The research adopt the generalizability theory to analyze the inter-rater reliability. Results For marking the complex ability, taking-out marking method was better（ estimated value of p, r, pr was 0. 743,0. 014,0.395 respectively）. The mark of less trained group （ Ep25 = 0.8519, Ф5 = 0.8519 ） was similar to the well trained group＇ s （ Ep25 = 0. 9039, Ф5 = 0.9010）, when adopting taking-out marking method to assess the complex ability. Conclusion The interaction between the method factors and the complexity of the ability may affect the inter-rater reliability.