知识增强型预训练语言模型综述

Review of Knowledge-Enhanced Pre-trained Language Models

知识增强型预训练语言模型旨在利用知识图谱中的结构化知识来强化预训练语言模型,使之既能学习到自由文本中的通用语义知识,又能够学习到文本背后的现实实体知识,从而有效应对下游知识驱动型任务。虽然该方向研究潜力巨大,但相关工作目前尚处初期探索阶段,并未出现全面的总结和系统的梳理。为填补该方向综述性文章的空白,在归纳整理大量相关文献的基础上,首先从引入知识的原因、引入知识的优势、引入知识的难点三方面说明了知识增强型预训练语言模型产生的背景信息,总结了其中涉及的基本概念;随后列举了利用知识扩充输入特征、利用知识改进模型架构以及利用知识约束训练任务等三大类知识增强方法;最后统计了各类知识增强型预训练语言模型在评估任务上的得分情况,分析了知识增强模型的性能指标、目前面临的困难挑战以及未来可能的发展方向。

The knowledge-enhanced pre-trained language models attempt to use the structured knowledge stored in the knowledge graph to strengthen the pre-trained language models,so that they can learn not only the general se-mantic knowledge from the free text,but also the factual entity knowledge behind the text.In this way,the enhanced models can effectively solve downstream knowledge-driven tasks.Although this is a promising research direction,the current works are still in the exploratory stage,and there is no comprehensive summary and systematic arrange-ment.This paper aims to address the lack of comprehensive reviews of this direction.To this end,on the basis of summarizing and sorting out a large number of relevant works,this paper firstly explains the background informa-tion from three aspects:the reasons,the advantages,and the difficulties of introducing knowledge,summarizes the basic concepts involved in the knowledge-enhanced pre-trained language models.Then,it discusses three types of knowledge enhancement methods:using knowledge to expand input features,using knowledge to modify model architecture,and using knowledge to constrain training tasks.Finally,it counts the scores of various knowledge enhanced pre-trained language models on several evaluation tasks,analyzes the performance,the current challenges,and possible future directions of knowledge-enhanced pre-trained language models.