摘要
本文从评分人决策的变化探讨了评分人个性化反馈信息的有效性。研究人员首先邀请了三位不同经验和背景的CET4作文评分人对30篇CET4模拟作文评分并提供三条评分理由,然后对另外10篇作文进行有声思维。评分结束一周后,评分人收到包括多层面Rasch模型(MFRM)的分析结果(严厉度、内在一致性和偏差)以及评分理由编码分析结果的个人反馈信息报告。阅读完反馈报告后,评分人接着对新30篇CET4模拟作文评分并对另外10篇作文进行有声思维(其中5篇作文和前测相同)。本研究通过对比分析反馈前、后相同的5篇作文的有声思维数据,结果发现反馈信息能帮助评分人重视评分标准相关特征,并调整自己的决策行为。
This study investigates the effectiveness of individualized feedback to raters from the perspective of the change in decision-making. First, three CET4 accredited essay raters with distinctive experience and background were invited to rate 30 mock essays, then to write and to rank order three reasons for their ratings simultaneously. Afterwards, they were trained on thinking aloud and then asked to rate 10 new essays while thinking aloud. One week later, raters received the feedback including results of the Multi-Facet Rasch Model (MFRM) analysis and coding analysis of reasons for their rating. Shortly after reading, raters were asked to rate a new set of 30 mock essays, and then rate another set of 10 essays (among which half are the same to the pretest) while thinking aloud. A detailed comparative analysis of raters' protocols before and after feedback on the 5 common essays indicated that the feedback is useful in helping raters derive scores with a heavier weight of rubric-related features, and make adjustments to their decision-making.
出处
《外语测试与教学》
2015年第1期1-11,共11页
Foreign Language Testing and Teaching
基金
广东省教育科学"十二五"规划项目"大规模考试中评分人决策风格的诊断研究"(批准号:2013JK013)
广州市社科规划青年项目"大规模语言测试评分人反馈信息有效性的实证研究--以CET4为例"(批准号:14Q11)
广东省高等学校教学质量与教学改革工程项目"大学英语课程多元评估体系建设与实践"(批准号:X2WY/N913078a)的阶段性研究成果