自然语言预训练模型知识增强方法综述

Knowledge Enhancement for Pre-trained Language Models:A Survey

将知识引入到依靠数据驱动的人工智能模型中是实现人机混合智能的一种重要途径。当前以BERT为代表的预训练模型在自然语言处理领域取得了显著的成功,但是由于预训练模型大多是在大规模非结构化的语料数据上训练出来的,因此可以通过引入外部知识在一定程度上弥补其在确定性和可解释性上的缺陷。该文针对预训练词嵌入和预训练上下文编码器两个预训练模型的发展阶段,分析了它们的特点和缺陷,阐述了知识增强的相关概念,提出了预训练词嵌入知识增强的分类方法,将其分为四类:词嵌入改造、层次化编解码过程、优化注意力和引入知识记忆。将预训练上下文编码器的知识增强方法分为任务特定和任务通用两大类,并根据引入知识的显隐性对其中任务通用的知识增强方法进行了进一步的细分。该文通过分析预训练模型知识增强方法的类型和特点,为实现人机混合的人工智能提供了模式和算法上的参考依据。

Introducing knowledge into data-driven artificial intelligence models is an important way to realize human-machine hybrid intelligence.The current pre-trained language models represented by BERT have achieved remarkable success in the field of natural language processing.However,the pre-trained language models are trained on large scale unstructured corpus data,and it is necessary to introduce external knowledge to alleviate its defects in determinacy and interpretability to some extent.In this paper,the characteristics and limitations of two kinds of pre-trained language models,pre-trained word embeddings and pre-trained context encoders,are analyzed.The related concepts of knowledge enhancement are explained.Four types of knowledge enhancement methods of pre-trained word embeddings are summarized and analyzed,which are pre-trained word embeddings retrofitting,hierarchizing the process of encoding and decoding,attention mechanism optimization and knowledge memory introduction.The knowledge enhancement methods of pre-training context encoders are described from two perspectives:1)task-specific and task-agnostic;2)explicit knowledge and implicit knowledge.Through the summary and analysis of the knowledge enhancement methods of the pre-trained language model,the basic pattern and algorithm are provided for the human-machine hybrid artificial intelligence.