面向问答系统的信息检索自动评价方法

Derive MAP-like metrics from content overlap for IR in QA system

对问答系统中的信息检索模块进行自动评价是开发问答系统中不可或缺的一环。采用传统的检索评价指标,就需要为每个问题标注正确的候选答案。为了避免这种代价,本文提出一种问答系统中的信息检索模块进行自动评价方法。该方法使用候选文档与问题本身以及问题参考答案间的信息,利用机器学习方法去拟合MAP。在实验中,本文发现使用GBDT模型拟合MAP值最好,斯皮尔曼等级相关系数达到了0.87。

During the development of a Question Answering System,implementing an automatic evaluation on the information retrieval module is an inevitable task. Using traditional information retrieval evaluation metrics requires labeling the correct candidate documents for each question. To avoid such cost,this paper proposes an automatic evaluation method for the information retrieval module in a Question Answering System. This method uses information between candidate documents and the question accompanied with the answer. This method utilizes machine learning technique to fit the MAP value. The experiment reviews that using GBDT model to fit the MAP value achieves the best performance and the corresponding Spearman rank correlation coefficient reaches. 7.