人工智能背景下机器评分与人工评分的效度比较——以英语学习者故事复述评分为例

Comparison of Validity of Machine Scoring and Validity of Manual Scoring Under the Background of Artificial Intelligence:Taking Scoring of EFL Learners’Story Retelling Performance as an Example

人工智能背景下,机器评分在口语测试领域中的应用得到了快速发展。基于R语言编写对英语学习者口头复述故事内容自动评分命令,进而比较机器(分析式)评分与人工(综合式和分析式)评分的效度,结果表明:人工评阅得分均值和机器评阅得分高度一致且无显著差异;人工分析式评阅得分均值和机器分析式评阅得分对学习者英语水平的解释力高于人工综合式评阅得分均值的解释力并具有共现效度;机器评分能够克服人工评分的主观性且节省人力和时间,可用于检验人工评分的信度;机器评分的局限性在于其准确性是建立在评分标准的完善和语音识别转录技术的优化基础之上的。

With the development of the technology of artificial intelligence,machine scoring has advanced rapidly in the field of oral testing.In order to compare the validity of machine analytic scoring with both human holistic and analytic scoring,the machine scoring R program for EFL learners’oral story retelling performance was written.The mean scores derived from two human scoring methods and the mean score from machine scoring are found to be highly consistent,and demonstrate no significant difference.Both human analytic and machine analytic scoring methods predict the learners’English proficiency more strongly than that of human holistic scoring,which proves their concurrent validity.Machine scoring can overcome the subjectivity of human scoring and save manpower and time.It can serve as a check for the reliability of manual scoring.However,one limitation of machine scoring lies in the fact that its accuracy is based on the improvement of scoring criteria as well as the optimization of speech recognition and transcription technology.