An easy-to-use evaluation framework for benchmarking entity recognition and disambiguation systems

实体识别消歧是知识库扩充和信息抽取的重要技术之一。近些年该领域诞生了很多研究成果,提出了许多实体识别消歧系统。但由于缺乏对这些系统的完善评测对比,该领域依然处于良莠淆杂的状态。因此很有必要设计一个评测框架对各个系统进行统一评测。本文提出一个实体识别消歧系统的统一评测框架,用于公平地比较各个实体识别消歧系统的效果。该框架代码开源,可以采用新的系统、数据集、评测机制扩展。通过该框架评测实体系统,可以分析得到系统各个模块的优劣之处。本文分析对比了几个公开的实体识别消歧系统,并总结出了一些有用的结论。

Entity recognition and disambiguation （ERD） is a crucial technique for knowledge base population and information extraction. In recent years, numerous papers have been published on this subject, and various ERD systems have been developed. However, there are still some confusions over the ERD field for a fair and complete comparison of these systems. Therefore, it is of emerging interest to develop a unified evaluation framework. In this paper, we present an easy-to-use evaluation framework （EUEF）, which aims at facilitating the evaluation process and giving a fair comparison of ERD systems. EUEF is well designed and released to the public as an open source, and thus could be easily extended with novel ERD systems, datasets, and evaluation metrics. It is easy to discover the advantages and disadvantages of a specific ERD system and its components based on EUEF. We perform a comparison of several popular and publicly available ERD systems by using EUEF, and draw some interesting conclusions after a detailed analysis.