科学基金项目立项评估:综合评价信息可靠性的多指标证据推理规则研究

Research Project Evaluation and Selection in the NSFC:An Approach Based on the Evidential Reasoning Rule with Reliability

公正合理的科研项目立项评估与选择是国家自然科学基金管理活动的关键环节。综合考虑科研项目的评估指标体系和选择流程,提出了利用历史评估准确性度量专家提供信息可靠性的方法,进而提出了一种系统性的基于证据推理规则的科学基金项目评估决策模型。该模型使用证据推理合成规则对多专家多指标评估信息进行集结。在集结过程中:充分考虑评估指标的权重以及评估等级的多样性;鉴于参与项目评估的专家具有不同的知识背景和经验,提出利用历史评估结果的准确性衡量专家提供的评价信息可靠性的方法;应用信度分布表征项目的整体评价结果,包含了更丰富的信息。国家自然科学基金项目评估的实例分析证明了该评估决策模型的有效性。

Research project evaluation and selection is one of the core research management activities in the National Science Foundation of China（NSFC）.Through taking into consideration evaluation criteria and selection procedure,a systematic method of implementing the evidential reasoning rule is presented to aggregate evaluation information on multiple criteria by multiple experts.The proposed method has the following strengths：1）combine evaluation information with different weights and reliabilities effectively,2）employ a confusion matrix for generating experts’ evaluation reliabilities,as experts may have different knowledge background and expertise,and 3）use a belief distribution to represent the overall performance profile of research projects in a more informative way.The belief distribution can be further transformed as utilities which are used to support the decision making of project selection.A case study on the project evaluation and selection of the National Science Foundation of China（NSFC）demonstrates the applicability and effectiveness of the proposed method.