大规模计算机口试分析评分效果研究

Analytic scoring for a computer-based speaking test

基于计算机的外语考试和评分越来越多,以往研究大都涉及基于纸笔评分效果,对基于计算机的主观题评分效果研究不多。本文从评分一致性和分数维度差异两方面研究了大规模计算机口试分析评分的效果。通过对比整体评分法发现,分析评分法的评卷员行为一致性较好。评卷员使用整体评分法时未能较好地根据表达内容的完整性打分,而且容易出现集中趋势。进行分析评分时,评卷员在内容和语言分项上一致性较差。在准确度方面,评卷员给低水平考生评分要比给高水平考生评分好。

Although computer-based rating of speaking tests is becoming more and more popular, relevant research on the reliability and validity of such rating is far from enough. This paper investigated the application of analytic scoring rubrics for a computer-based speaking test. Scoring consistency and variability of candi-dates' profiles of analytic scores were discussed by means of comparing the results of analytic scoring and holistic scoring. Results showed that the analytic rating scale outperformed the holistic one. The raters using both scales failed to give expected ratings in evaluating the content of the students ' performance, though they demonstrated acceptable reliability in rating the students' pronunciation and fluency. Information functions also revealed certain degree of rating inaccuracy in assessing the performance of the high-level students. Implications of this study were discussed and suggestions were given in terms of rater training and choice of rating scales.