Automated Chinese Essay Scoring Based on Deep Learning

Writing is an important part of language learning and is considered the best approach to demonstrate the comprehensive language skills of students.Manually grading student essays is a time-consuming task;however,it is necessary.An automated essay scoring system can not only greatly improve the efficiency of essay scoring,but also provide more objective score.Therefore,many researchers have been exploring automated essay scoring techniques and tools.However,the technique of scoring Chinese essays is still limited,and its accuracy needs to be enhanced further.To improve the accuracy of the scoring model for a Chinese essay,we propose an automated scoring approach based on a deep learning model and validate its effect by conducting two comparison experiments.The experimental results indicate that the accuracy of the proposed model is significantly higher than that of multiple linear regression(MLR),which was commonly used in the past.The three accuracy rates of the proposed model are comparable to those of the novice teacher.The root mean square error(RMSE)of the proposed model is slightly lower than that of the novice teacher,and the correlation coefficient of the proposed model is also significantly higher than that of the novice teacher.Besides,when the predicted scores are not very low or very high,the two predicted models are as good as a novice teacher.However,when the predicted score is very high or very low,the results should be treated with caution.

ChatGPT与AES系统对大学英语写作的反馈效度比较

以45篇非英语专业大学生的英语作文为样本,比较了ChatGPT和国内典型3个作文自动评分系统(AES)的评分效度,并通过案例分析比较ChatGPT和AES系统代表批改网在整体评价、纠错反馈、篇章结构和观点合理性评估能力上的差异。研究发现,ChatGPT和3个AES系统评分平均都低于教师评分,且ChatGPT的评分最低;ChatGPT与教师评分结果呈中等正相关,但它与教师评分相邻吻合一致性不如3个AES系统。ChatGPT在内容反馈上的优势较为明显,其整体评价更加全面、个性化,错误识别更加准确,修改建议更加直接,对篇章结构和观点内容的判断能力显著优于AES系统,但也存在评分不稳定、提示语理解偏差和“类人化”惰性表现等局限性。通过了解ChatGPT的优势和局限性,有利于其更科学地赋能英语写作教学。

Proposal of Japanese Vocabulary Difficulty Level Dictionaries for Automated Essay Scoring Support System Using Rubric

We are developing a Moodle plug-in,which is an AES(automated essay scoring)support system for the basic education of university students.Our system evaluates essays based on rubric,which has five evaluation viewpoints“Contents,Structure,Evidence,Style,and Skill”.Vocabulary level is one of the scoring items of Skill.It is calculated using Japanese Language Learners’Dictionaries constructed by Sunakawa et al.Since this does not fully cover the words used in the student-level essays,we found that there is a problem with the accuracy of the vocabulary level scoring.In this paper,we propose to construct comprehensive Japanese vocabulary difficulty level dictionaries using Japanese Wikipedia as the corpus.We apply Latent Dirichlet Allocation(LDA)to the Wikipedia corpus and find the word appearance probability as one of the indexes of word difficulty.We use the TF-IDF value instead of the LDA value of the words,which rarely appears.As a result,we constructed highly comprehensive Japanese vocabulary difficulty level dictionaries.We confirmed that the vocabulary level can be scored for all words in the test dataset by using the constructed dictionaries.

大学英语作文自动评分中分级词表的效度研究

自动作文评分中的核心问题是对高信度的、机器可用的评分特征的选取。大学英语作文自动评分中的一个基本特征是词汇分布，而词汇分布的衡量需要一个描述清晰、刻画准确的大学英语作文分级词表，也就是一个效度较高的词表。实验证明目前已有的词表难以达到这个要求，但通过改进或者调整，可以取得对不同作文质量效度较高的词表。

基于小数据集抽取算法SDRA的主观题自动评分研究

主观题自动评分(Automated Essay Scoring,AES)是语言测试领域的难点,自动评分的第一步往往要用大量已经人工评分的数据集生成评分模型,但是研究者常常困顿于数据集的不足。从数学的角度来看,其实只需要小数据集就可以构建出媲美大数据集所生成的评分模型,从而有效提高科研工作者的效率和数据利用率。本文提出一种小数据集抽取算法(Small Dataset Retrieval Algorithm,SDRA),在基于潜伏语义分析方法(LSA)的自动评分软件平台上,分别实施SDRA和传统的大数据集的评分模型方法,实验结果表明,SDRA仅用了25%的数据量,即可达到接近于传统大数据集的评分效果,从而验证了SDRA是行之有效的。

基于多层次语义特征的英文作文自动评分方法

作文自动评分(AES)技术能够自动地对作文进行分析和评分,其已成为自然语言处理技术在教育领域应用的热点研究问题之一。针对目前AES方法割裂了深层和浅层语义特征,忽视了多层次语义融合对作文评分影响的问题,提出了一种基于多层次语义特征的神经网络(MLSF)模型进行AES。首先,采用卷积神经网络(CNN)捕获局部语义特征,并采用混合神经网络捕获全局语义特征,以从深层次获取作文的语义特征;其次,利用篇章级的作文主题向量来获取主题层特征,同时针对深度学习模型难以挖掘的语法错误和语言丰富程度特征,构造了少量人工特征以从浅层获取作文的语言学特征;最后,通过特征融合对作文进行自动评分。实验结果表明,所提出模型在Kaggle ASAP竞赛公开数据集的所有子集上性能均有显著提升,该模型的平均二次加权的卡帕值(QWK)达到79.17%,验证了该模型在AES任务中的有效性。

HSK自动作文评分的特征选取研究

作文特征选取是研究汉语作为第二语言的水平测试自动作文评分的关键问题之一,以中国汉语水平考试作文为研究对象,从字、词、语法、成段表达、庄雅度等多个层面上,选取107个作文特征,经相关度计算得到19个与作文分数较为相关的作文特征。基于选取的作文特征,采用多元线性回归方法进行回归实验和稳定性交叉实验。实验表明,作文长度、词汇使用和成段表达方面的作文特征对作文得分具有较好的解释能力,多元线性回归方法应用于中国汉语水平考试自动作文评分具有较好的稳定性。

The Impact of Online Automated Writing Evaluation:A Case Study from Dalian

Automated writing evaluation （AWE） technology is being adopted in classrooms in China and the USA. This paper presents the results of a case study of the application of AWE in Dalian, China. The quasi-experimental study was conducted in 2010 in Chinese middle school English language classrooms. An effect size of 0.30 was found in favor of the experimental group using AWE as an online formative assessment. Student survey responses and teacher observations are presented as convergent evidence to illustrate the impact AWE technology has on teachers, students, and student achievement in English writing.

自动评分系统在二语写作教学中的反馈效果实证研究

教学实践和研究实验发现作文自动评分系统(AES)对于提高学生的二语写作能力发挥着比较积极的作用。本文以句酷批改网为例,探究学生利用自动评分系统多次修改后所提交作文,在词汇、句法、结构、规范四大维度上的变化,研究发现AES对于学生英语写作词汇、写作规范的影响较大,这其中又以学术词汇密度、高频词汇密度、类符形符比以及拼写正确率和病句数的变化最为明显。因此,作文自动评分系统对于学生二语写作的反馈是及时且有效的。研究最后针对现有的AES不足之处提出了几点建议。