摘要
评分人反馈信息是保证评分质量的一种重要手段,然而以往的研究对该方法的有效性尚无定论。本研究邀请三位评分人对30篇CET4模拟作文评分,同时提供按重要性排列的三条评分理由。评分结束一周后,评分人收到个性化反馈信息报告,内容包括多层面Rasch模型(MFRM)的分析结果(严厉度、内部一致性和偏差)以及评分理由的编码分析结果。阅读完反馈信息后,评分人对另外30篇CET4模拟作文评分并提供评分理由。研究发现,反馈信息能加深评分人对评分标准的理解,提高评分人的内部一致性,减少评分理由中的构念不相关差异。
Feedback to raters is widely used as a common practice to control rating quality, while previous studies have produced mixed findings. This paper inquires into the effectiveness of individualized rater feedback by inviting three CET4 accredited raters to rate 30 mock essays, then write and rank three reasons for their ratings simultaneously. One week later, raters received detailed individualized feedback report of their performance including results of the MFRM analysis and coding analysis for their rating reasons. Raters were asked to attend to the feedback when marking a new pack of 30 mock essays, then write and rank three reasons as well. A comparison between rater performance before and after feedback revealed that raters were able to gain a deeper understanding of rating scale, become more self-consistent and decrease construct-irrelevant variance.
出处
《天津外国语大学学报》
2014年第1期62-69,共8页
Journal of Tianjin Foreign Studies University
