基于数据的故事复述评分标准的效度验证

Validating an Empirically Developed Rating Scale of Story Retelling Task

本研究采用Knoch(2009)评分标准效度验证框架,从信度、构念效度和实用性3方面收集证据对基于数据的故事复述评分标准进行了效度验证。定量数据包括5位评分员评60份考生录音的分数,定性数据包括他们提供的评分理由和评分后的回溯性访谈。定量分析结果表明:该评分标准区分度较好;虽然评分标准3个维度的难度具有显著差异,但不存在非拟合或过度拟合的情况。定性分析结果表明:评分员能准确理解并合理运用该评分标准进行评分,不存在评分理由和评分标准不相关的情况;他们认为该评分标准可操作性强,能充分表征任务构念。

Within the framework of facets of rating scale validity(Knoch 2009),this study aims to validate an empirically developed rating scale of the story retelling task by collecting evidence in terms of reliability,construct validity and practicality.The data consists of five raters'ratings of 60 test-takers'voice recordings,their reasons for the ratings and interviews of their perceptions of using the rating scale.Quantitative analysis reveals that the rating scale can discriminate test-takers effectively.It has neither misfit nor overfit dimension despite the existence of significant differences between three dimensions in difficulty.Qualitative analysis results show that raters can understand the rating scale accurately and use it appropriately as a guide in the rating practice.There is no scale-irrelevant variance in their ratings,and they perceive the scale as representing the construct adequately and being practical.