视觉语言预训练综述

Survey on Vision-language Pre-training

近年来深度学习在计算机视觉(CV)和自然语言处理(NLP)等单模态领域都取得了十分优异的性能.随着技术的发展,多模态学习的重要性和必要性已经慢慢展现.视觉语言学习作为多模态学习的重要部分,得到国内外研究人员的广泛关注.得益于Transformer框架的发展,越来越多的预训练模型被运用到视觉语言多模态学习上,相关任务在性能上得到了质的飞跃.系统地梳理了当前视觉语言预训练模型相关的工作,首先介绍了预训练模型的相关知识,其次从两种不同的角度分析比较预训练模型结构,讨论了常用的视觉语言预训练技术,详细介绍了5类下游预训练任务,最后介绍了常用的图像和视频预训练任务的数据集,并比较和分析了常用预训练模型在不同任务下不同数据集上的性能.

In recent years,deep learning has achieved excellent performance in unimodal areas such as computer vision(CV)and natural language processing(NLP).With the development of technology,the importance and necessity of multimodal learning begin to unfold.Essential to multimodal learning,vision-language learning has received extensive attention from researchers in and outside China.Thanks to the development of the Transformer framework,more and more pre-trained models are applied to vision-language multimodal learning,and the performance of related tasks is improved qualitatively.This study systematically reviews the current work on vision-language pretrained models.Firstly,the knowledge about pre-trained models is introduced.Secondly,the structure of pre-trained models is analyzed and compared from two perspectives.The commonly used vision-language pre-training techniques are discussed,and five downstream pretraining tasks are elaborated.Finally,the common datasets used in image and video pre-training tasks are expounded,and the performance of commonly used pre-trained models on different datasets under different tasks is compared and analyzed.