从视频到语言:视频标题生成与描述研究综述

From Video to Language:Survey of Video Captioning and Description

视频标题生成与描述是使用自然语言对视频进行总结与重新表达.由于视频与语言之间存在异构特性,其数据处理过程较为复杂.本文主要对基于“编码−解码”架构的模型做了详细阐述,以视频特征编码与使用方式为依据,将其分为基于视觉特征均值/最大值的方法、基于视频序列记忆建模的方法、基于三维卷积特征的方法及混合方法,并对各类模型进行了归纳与总结.最后,对当前存在的问题及可能趋势进行了总结与展望,指出需要生成融合情感、逻辑等信息的结构化语段,并在模型优化、数据集构建、评价指标等方面进行更为深入的研究.

The task of video captioning and description is to summarize and re-express the visual content of video with natural language/text.It is challenging because it involves the transformation of different modal information,and there exists heterogeneity between the visual data and language.In this work,the models based on the“encoder-decoder”pipeline are mainly elaborated in detail.According to the encoding and usage of visual features,the current models are classified into four types:the models based on mean/max pooling feature,the models based on video sequential memory,the models based on 3D CNN feature,and the models based on hybrid features.A number of popular works of each type are described and analyzed.Finally,the existing problems and possible trends worth studying are summarized.It is pointed out that the prior knowledge including emotion and logical semantics in complex videos should be further mined and embedded for the generation of logical paragraph description.Moreover,it is still desired to further investigate the techniques of model optimization,dataset construction and evaluation metrics for video captioning and description.