同行评议一致性研究

A Study of the Agreement of Peer Review

评分一致性是衡量同行评议评价效果的重要指标之一。为了探究同行评议中评审人在研究质量评价上的一致性,本研究基于ICLR会议的公开同行评议数据,使用多位评审人对一项研究评分的方差衡量一致性,并采用单因素方差分析等方法,分析了对不同研究质量论文评审人评分一致性的差异。结果发现,同行评审对于质量较差论文的一致性高于质量较好的论文。基于这一证据,本研究提出以"择差"为目的的"底线评审"的构想,以完善当前以"择优"为宗旨的同行评议制度。

The agreement of reviewers’ratings is one of the important aspects to measure the effectiveness of peer review evaluation.In order to explore the agreement of peer review in evaluating research quality,this study conducted an empirical analysis based on the open peer review data of the ICLR conference.We quantified the degree of the agreement as the variance of reviewer’s ratings on the evaluation of a single paper,and used one-way ANOVA to analyze the differences in the degree of consensus of reviewers on the papers of each score level.The results show that peer review has a higher degree of agreement on lowquality articles than high-quality articles.The conclusion of this study can be also extended to the review of research funding and provide theoretical basis for the evaluation strategy of the newly proposed“bottom-line review”.