写作测试内容质量评分研究——分层决策树法

Assessing content quality of writing performance: The Hierarchical Decision-Tree Approach

本研究旨在开发一项针对写作测试内容质量的评分方法——分层决策树法,从理论和实证两个层面考察分层决策树法的评分质量。两名评分员分别使用分层决策树法和整体评分法对北京某高校81名英语专业学生写作测试的内容质量进行评分。研究结果表明:1)分层决策树法能够有效区分不同英语水平的考生组别;2)相关分析的结果显示,内容质量同所选写作特征是互相影响和关联的,呈现不同程度的相关关系;3)相比传统整体评分,分层决策树评分法的评分员信度较高,并且分层决策树法的重测评分信度较高。配对样本t检验的结果也表明分层决策树法作为内容质量的评分方法具有稳定性。

This paper reports an empirical study of a new rating method——the Hierarchical Decision-Tree Approach for assessing content quality in writing assessment. The new approach was compared with traditional holistic scoring approach by two experienced raters assessing 81 writing scripts of English major students. Results show that:(1) the proposed approach could significantly differentiate the students at three proficiency levels;(2) the selected features are correlated to content quality measured by the new approach to different extent; and (3) the proposed approach enjoy relatively higher inter-rater reliability and test-retest reliability compared with those of traditional scoring. The results of paired sample t-test also confirmed the stability of the new approach.