管理科学部经济科学学科人工智能指派与分类评审效果分析

The Effect of Using AI Assignment and Category-specific to the Review of National Natural Science Fund:Piloting by the Economic Science

本文对国家自然科学基金委员会管理科学部经济科学学科(G03)2022年面上项目和青年科学基金项目人工智能指派和分类评审的效果进行分析。基于在经济科学学科国际经济与贸易领域的随机对照试验发现,人工智能指派能够高效匹配“小同行”开展通讯评审,有效提升评审项目共识度和上会率。经统计检验,人工智能指派实验组和对照组样本的项目评审得分分布与均值没有显著差异,意味着人工智能指派不会系统性地影响同行通讯评审结果。2022年经济科学学科全面开展基于科学问题属性的分类评审,统计数据表明分类评审能够有效使同行评议专家按科学问题属性形成共识,明显缩小原创、前沿、交叉类项目同需求类项目之间上会率的差异。

This study evaluates the effect of the AI assignment and category-specific review in 2022,which covers the General Program and the Youth Scientists Fund in Economic Science(G03)of Division III in the Department of Management Sciences.Based on the randomized controlled trials in the field of International Economics and Trade,we find that the AI assignment can effectively match experts to carry out peerreview.With AI assignment,the experts'consensus on applications has been improved,which increases the probability of being on the conference review.With statistical tests,there is no significant difference in the mean value of the scores between the treatment and control groups,which implies that the AI assignment has not systematically affected the results of the peer-review.All applications in Economic Science(G03)have been under category-specific review based on scientific problem attributes this year.The statistics again show that category-specific review can effectively enable experts to form consensus,and significantly reduce the difference in the rate for the applications being on the conference review among the Original Exploratory program,the Cutting-edge program,the Transdisciplinary program,and the Application program.