基于层次分析法的科技项目同行评议专家综合评价体系构建研究

Research on the Establishment of Comprehensive Evaluation System for Peer Review Experts of Science and Technology Projects Based on the Analytic Hierarchy Process

同行评议广泛应用于科研项目的评审,是科研项目评审的生命线。随着各类科研项目申请数量的增加及学科体系的细化,同行评议方法的局限性逐渐凸显。评议专家是同行评议的基石,对评议专家进行反评估可实现对专家的有效监督,进一步保证评议结果的公正客观。针对省级科技计划项目的特点,利用层次分析法构建同行评议专家综合评价体系对专家基本能力(个人资历、科研能力)和评审绩效(评审离散率、评审有效率、评审态度)进行评价,可实现对专家的定量反评估。借鉴国外成熟的申请者反馈机制,在所构建的评价体系中增设了申请者反馈情况指标;同时,将评审有效率指标中的成功率指标与项目验收评价等级关联,构建了验收评价打分指标体系。基于收集的评审实例数据,对评价体系进行了实证分析。经验证,该评价体系可有效评估同行评议专家的评审质量,可为后续针对评议专家的监督约束、信誉积累和循证问责等提供数据支撑,亦可为专家数据库的动态管理提供决策参考。

Peer review is widely used in the review process of science research projects, which is of great importance and can be regarded as its lifeline without any exaggeration. With the rapid increase in the number of applications for various science research projects and the refinement of the discipline system, the limitations of peer review methods are gradually highlighted. It is known that the experts are the cornerstones as far as peer review is concerned. Establishing a scientific and reasonable evaluation system for experts themselves is the premise of effective supervision and regulation of the peer review process, ensuring the objectiveness and fairness of the final assessing results. According to the characteristics of science and technology projects, a comprehensive evaluation system was constructed based on the analytic hierarchy process. Then the quantitative meta-evaluation of experts could be achieved by assessing their personal qualifications and scientific research abilities, and previous review status including review dispersion rate, review efficiency, review attitude and so forth. Referring to the mature mechanism of applicant feedback extensively used in many developed countries, a new indicator of applicant feedback was added into the aforementioned evaluation system. Moreover, a scoring index system of acceptance evaluation was established by associating the acceptance evaluation level with the success rate index. According to the results of one kind of local-provincial scientific research project investigated in previous years, the validity and efficiency of this system on the review quality of expert was well verified. This can provide the data to support the supervision and restraint, reputation accumulation and evidence-based accountability of the experts in the future. On the other hand, it can also act as decision-making reference for dynamic management of the expert database.