英语作文自动评分系统的信度和效度研究——基于不同类型写作任务文本量化特征分析

A study on the reliability and validity of AWE system for English writing—A quantitative analysis of different types of writing tasks

本研究使用L2 Lexical Complexity Analyzer、L2 Syntactic Complexity Analyzer及Coh-Metrix软件分析了826篇不同类型写作任务的大学英语六级应试作文在词汇复杂性、句法复杂性及语篇连贯性层面的量化指标,结合言语失误特征构建其各自的打分模型,并针对进入模型的预测变量进行对比分析,探究英语自动评分系统对不同类型写作任务评阅的信度和效度。研究显示:自动评分系统对现象观点型作文的评分效度高,而对名言警句型作文的评阅效度较差。作文评阅过程中,系统更关注写作规范、高级词汇、长句、复杂句及其句式的变化等,而严重淡化句法错误、语篇意义连贯性等文本特征,这无疑会对作文评阅的质量造成不利影响。本研究结果不仅为教师网批作文提出很好的建议,也为优化自动评分系统,助力英语写作教学提供有益参考。

This study quantitatively analyzed the lexical and syntactic complexity,coherence and error pattern of 826 timed essays by L2 Lexical Complexity Analyzer,L2 Syntactic Complexity Analyzer and Coh-Metrix,intending to establish the rating model of the Automatic Writing Evaluation(AWE)system for different types of writing tasks and to explore its reliability and validity.The result shows that AWE system demonstrates higher validity in dealing with phenomenal viewpoint writing than epigram elaboration writing;and it is found to be very sensitive to writing norms,vocabulary complexity and sentence complexity,length and diversity,while seriously downplaying textual features such as syntactic errors and discourse coherence which will have a negative effect on the reliability and validity of writing evaluation.The results of the study provide helpful suggestions not only for evaluating writings online,but also for optimizing AWE system and improving English writing teaching.