批改网和iWrite评分信度及效度对比研究

A Comparative Study on Reliability and Validity of Assessments by Pigai and iWrite

随着国内外自然语言处理、人工智能等技术的不断发展,各种英语作文自动评分系统层出不穷,以批改网与iWrite为代表的机器评分在大学阶段得到广泛的应用。文章以英语专业大四学生的英语作文为研究对象,信度分析发现批改网、iWrite评分信度较高,但是均分都显著低于人工评分。批改网与iWrite无法从作文的逻辑结构、连贯性等深层维度准确评分,效度都不高。但是这两种评分系统存在各自的优势,批改网从词汇、词语搭配、语法等语言层面提供反馈,iWrite根据文体对学生作文质量提供整体反馈。教师根据教学目的,择优而用之,辅助英语作文教学。

With the development of the processing of natural languages,artificial intelligence and other technologies at home and abroad,various automatic scoring systems for English writing have emerged one after another.The English writings by senior English majors are taken as the research object.The reliability test shows that scores of Pigai and iWrite are reliable but are significantly lower than the scores by human raters.Pigai and iWrite can’t accurately evaluate the logical structure of the writings and their coherence.Two systems are not valid in some degree.However,Pigai and iWrite have their own advantages.Pigai provides feedbacks on the linguistic level,such as vocabulary,collocations and grammar while iWrite directly provides overall feedbacks on the quality of students’writings.Teachers can choose either scoring system to assist the English writing teaching according to the teaching purposes.