ChatGPT支持的学生论证内容评价与反馈——基于两种提问设计的实证比较

Effectiveness of Feedback on Students’Argu mentation Contents Based on ChatGPT:Comparison of Two Types of Prompt Designs

在论证式教学中,由于学生生成的论证内容量大且复杂,教师评价与反馈往往滞后且难以保证质量。生成式人工智能聊天工具ChatGPT的出现为解决该问题提供了可能。与ChatGPT互动的质量取决于提问设计,如何向其提问成为获得有效反馈的关键。基于“初始提问”和“优化提问”两种提问设计,利用ChatGPT对50份学生论证内容进行评价与反馈,从反馈精准度和反馈类型两方面对其效果展开实证比较发现:“优化提问”下ChatGPT的反馈精准度(含精确度和召回率)高于“初始提问”,但在两种提问设计下的反馈召回率均低于精确度,且在量化评价维度上的精准度表现优于质性评价维度;基于两种提问设计,ChatGPT均能针对论证内容生成任务型反馈、过程型反馈、建议型反馈和情感型反馈,但相较于“初始提问”,其基于“优化提问”所生成的反馈更具组织性、解释性和针对性,而两种提问设计下的情感型反馈均表现出“就事论事”“中庸”的特点。为有效发挥ChatGPT在教学评价与反馈中的潜能,教师需做好提问设计,发挥其在情感反馈上的优势,并对机器反馈进行把关,同时注重培育学生的反馈素养。

In argumentative teaching,due to the large volume and high complexity of students’argumentation contents,teachers’evaluation and feedback often lag behind and are difficult to ensure the quality.The emergence of such chat tools based on generative artificial intelligence as ChatGPT provides the possibility to solve this problem.The quality of interaction with ChatGPT depends on the prompt design.How to design prompts becomes the key to obtaining effective feedback.Based on two types of prompt designs(“initial prompts”and“optimized prompt”),ChatGPT was used to evaluate and provide feedback on 50 copies of students’argumentation contents.Empirical comparison was conducted on its effectiveness from two aspects:feedback accuracy and feedback type.It was found that ChatGPT’s feedback accuracy(including precision rate and recall rate)under“optimized prompt”was much higher than that under“initial prompt”.The feedback’s recall rates under both types of prompt designs were lower than corresponding precision rates.The feedback’s precision rates of the quantitative evaluation dimension were higher than those of the qualitative evaluation dimension under both types of prompt designs.Based on the two types of prompt designs,ChatGPT could generate four kinds of feedback for argumentation contents:task-oriented feedback,process-oriented feedback,suggestion-oriented feedback,and emotion-oriented feedback.However,compared to feedback based on the“initial prompt”,the feedback generated based on“optimized prompt”was more organized,explanatory,and targeted.The emotion-oriented feedback under both types of prompt designs exhibits such characteristics as“confining merely to the facts”and“the doctrine of the mean”.To effectively unleash the potential of ChatGPT in teaching evaluation and feedback,teachers need to improve their prompt designs,leverage their advantages in emotion-oriented feedback,monitor feedback from ChatGPT and focus on cultivating students’feedback literacy.