评价量规设计对慕课同伴互评有效性的影响研究

Study on the Effectiveness of Design of Assessment Rubric for Peer Assessment in MOOC

同伴互评是MOOC常采用的学员作业评价方法,但是MOOC同伴互评的准确性和可靠性因学员知识水平程度差异过大而被质疑。为了研究如何通过改进评价量规来提高同伴互评的可靠性和准确性,在教师培训类MOOC"翻转课堂教学法"的两期教学中,我们进行了若干设计研究。研究发现,在教学培训类MOOC的同伴互评活动中,学生评分和教师评分高度相关。细化评分量规的等级描述,以及采用是非选择型的评价量规都能显著提高评分者间信度和评分效度,且评分者间信度与评分效度的差异度与学生所提交作业文档本身的质量也有关系。在研究发现的基础上,文章提出了MOOC同伴互评评价量规的设计原则和使用建议。

Peer assessment is an evaluation method for students＇ assignments commonly used in MOOCs.However, the reliability and validity of peer assessment are questioned because of the different levels of students. In order to study how to improve the reliability and validity of peer assessment by improving the rubrics, we conduct a number of design studies in the two-phase teacher training MOOC ＂flipped classroom instruction＂. The study shows that there is a high correlation between student ratings and teacher ratings in the peer assessment of teaching training MOOC. Refining the level descriptions of the rubrics or using true-false judgement as a criterion can significantly improve the inter-raters＇ reliability and validity. Moreover, the difference of inter-raters＇ reliability and validity is influenced by the quality of students＇ assignments. Finally,this paper proposes the design principles and some suggestions for the rubrics of MOOC peer assessment.