摘要
BERT模型是一种全新的语言模型,其采用微调的双向Transformer编码实现,利用fine-tuning的方式进行预训练,实际使用时,只需要根据具体任务额外增加一个输出层就可以用来解决特定任务,克服了传统词嵌入模型对不同任务定义不同网络结构的缺点。为了更好地理解BERT模型及其效果,首先概述BERT模型的原理,以及BERT的预训练策略,然后介绍了如何将BERT模型应用于3个下游任务:文本分类、机器阅读理解和文本摘要,并通过对比实验展示了BERT模型的优势。最后,对未来研究方向进行了展望。
The BERT model is a new language model,which is implemented by fine-tuned bidirectional Transformer coding.Using Fine-Turning method to get the pre-trained model,it only needs to add an additional output layer according to specific tasks,then it can be used to solve the tasks,thus overcoming the shortcomings of the traditional word embedding model which needs to define different tasks for different network structure.In order to understand the BERT model and its effect better,the principle of the model is reviewed,the pretrain strategy of the model is introduced in this article.Then how to apply the BERT model to the three downstream tasks:text classification,machine reading comprehension,and text summary are introduced,and the advantages of the BERT model through comparative experiments are demonstrated.Finally,the future research direction is prospected.
作者
段瑞雪
巢文宇
张仰森
DUAN Ruixue;CHAO Wenyu;ZHANG Yangsen(Computer School,Beijing Information Science&Technology,Beijing 100192,China;School of Information Management,Beijing Information Science&Technology,Beijing 100192,China;Beijing Laboratory of National Economic Security Early Warming Project,Beijing 100044,China)
出处
《北京信息科技大学学报(自然科学版)》
2020年第6期77-83,共7页
Journal of Beijing Information Science and Technology University
基金
北京市自然科学青年基金项目(4204100)
北京信息科技大学校基金(1825023)
北京信息科技大学2020年促进高校内涵发展-大学生科研训练项目(5102010805)
北京信息科技大学2019年度‘实培计划'项目资助。
关键词
预训练
机器阅读理解
文本分类
文本摘要
pre-train
machine reading comprehension
text classification
text summarization