基于电子档案袋测评的评分者间信度分析报告

An examination of inter-rater reliability of an e-portfolio assessment

本研究旨在评估一项电子档案袋测评实验的评分者间信度,分析评分差异的可能原因。从样本框按学号抽取85名学生的档案袋(28.3%)。8位评分者分成5组(每组2人,1名任课教师,1名外部评分者),根据量规对旨在反映学生反思能力和自我测评能力的两指标进行独立评分。分别用Cohen的kappa系数、PABAK指数、Spearman秩相关系数、配对样本t检验和效应量对两个评分者的一致性、相关性和差异性进行估计。结果表明:1)10组评分中,7组kappa值达到中等以上一致,9组修正kappa-PABAK值达到中等以上一致;2) 10组评分均达到中等到极强相关程度;3)任务类型和任务呈现方式、评分者角色以及对量规的理解一定程度上造成了评分偏差。本文针对研究结果讨论了对未来档案袋测评设计的启示。

This study examines the inter-rater reliability of teachers’ scoring in an e-portfolio assessment experiment.A sample of 85 student portfolios are chosen from the sampling frame according to their ID number(28.3%).Eight raters in 5 pairs(one is the course teacher and the other external rater) independently score students’ performances against the rubric on two tasks that aim to develop students’ reflective skills and self-assessment skills.Inter-rater agreement,correlation as well as differences are estimated using Cohen’s kappa,PABAK,Spearmans’ s rank coefficient,paired sample t test and effect size.Results show that:1) a moderate to substantial agreement could be found in 7 rater-pairs if kappa values are used,and in 9 of the 10 rater-pairs in the PABAK values;2) a moderate to very strong level of correlation existes in the 10 rater-pairs;and 3)task types and structure,rater’s role,raters’ understanding of the rubric or the course contribute to the rater differences.This article concludes with implications for future research on portfolio design for assessment purpose.