创造力测评中的评分者效应

Rater effects in creativity assessment

创造力测评中的评分者效应(ratereffects)是指在创造性测评过程中,由于评分者参与而对测评结果造成的影响。评分者效应本质上源于评分者内在认知加工的不同,具体体现在其评分结果的差异。本文首先概述了评分者认知的相关研究,以及评分者、创作者、社会文化因素对测评的影响。其次在评分结果层面梳理了评分者一致性信度的指标及其局限,以及测验概化理论和多面Rasch模型在量化、控制该效应中的应用。最后基于当前研究仍存在的问题,指出了未来可能的研究方向,包括深化评分者认知研究、整合不同层面评分者效应的研究,以及拓展创造力测评方法和技术等。

Rater effects refer to the impact of different raters’ idiosyncrasies in their behaviors on the evaluation results in creativity assessment. Rater effects are due to the difference in raters’ cognitive process of the evaluation, which are externally reflected in the difference of their scorings. This article first summarizes the studies of rater cognition and other influencing factors on creativity assessment, including characteristics of raters, information of creators and socio-cultural factors. It further examines inter-rater reliability indexes and their limitations, as well as the applications of Generalization Theory and Many-Facet Rasch Model in quantifying and controlling of rater effects. Finally, this paper specifies directions of future research based on the existing limitations, including deepening the investigation on rater cognition in creativity assessment, integrating the studies of rater effects on different levels, and developing new methods and techniques of creativity assessment.