自然语言处理中的预训练范式

Pre-training Paradigm in Natural Language Processing

从2017年以来,自然语言处理中提出了“预训练+微调+师生学习”的新范式。这种新范式是自然语言处理的第四代范式,它代表着自然语言处理未来发展的方向。本文讨论自然语言处理中的这种新范式,分别介绍迁移学习、预训练、微调的原理和方法,并说明它们在Transformer、BERT、UniLM和Unicoder等模型中的应用。

Since 2017,the“pre-training+fine-tuning+teacher-student-learning”paradigm is proposed in natural language processing(NLP).This new paradigm is the fourth generation paradigm in NLP,and it represents a new development direction of NLP.The paper discusses this paradigm in NLP,introduces the principles and methods of transfer learning,pre-training and fine-tuning,and explains the application of these principles and methods in the models of Transformer,BERT,UniLM and Unicoder.