知识指导的预训练语言模型

Knowledge-Guided Pre-Trained Language Models

作为典型的数据驱动工具,预训练语言模型(PLM)仍然面临可解释性不强、鲁棒性差等难题。如何引入人类积累的丰富知识,是改进预训练模型性能的重要方向。系统介绍知识指导的预训练语言模型的最新进展与趋势,总结知识指导的预训练语言模型的典型范式,包括知识增强、知识支撑、知识约束和知识迁移,从输入、计算、训练、参数空间等多个角度阐释知识对于预训练语言模型的重要作用。

As a typical data-driven method,pre-trained language models(PLMs)still face challenges such as poor interpretablility and robust⁃ness.Hence,it is important to introduce human knowledge into these models for better performance.The latest progress and trend of knowledge-guided PLMs are introduced and the paradigm of knowledge-guided PLMs is summarized,including knowledge augmentation,knowledge support,knowledge regularization,and knowledge transfer.