基于多模态特征的视频密集描述生成方法

A Dense Video Captioning Method Based on Multi-modal Features

根据视频内容自动生成文本序列的密集描述生成融合了计算机视觉与自然语言处理技术。现有密集描述生成方法多强调视频中的视觉与运动信息而忽略了其中的音频信息,关注事件的局部信息或简单的事件级上下文信息而忽略了事件间的时序结构和语义关系。为此,该文提出一种基于多模态特征的视频密集描述生成方法。该方法首先在动作提议生成阶段使用Timeception层作为基础模块以更好适应动作片段时间跨度的多样性,其次在动作提议生成和描述生成两阶段均利用音频特征增强提议和描述生成效果,最后使用时序语义关系模块建模事件间的时序结构和语义信息以进一步增强描述生成的准确性。特别地,该文还构建了一个基于学习场景的视频密集描述数据集SDVC以探究该文所提方法在学习场景现实应用中的有效性。在ActivityNet Captions和SDVC数据集上的实验结果表明,动作提议生成AUC值分别提升0.8%和6.7%;使用真实动作提议进行描述生成时,BLEU_3值分别提升1.4%和4.7%,BLEU_4值分别提升0.9%和5.3%;使用生成的动作提议进行描述生成时,SDVC数据集BLEU_3、BLEU_4值分别提升2.3%和2.2%。

Dense video captioning can automatically generate sentence sequence corresponding to video content,involving issues in both computer vision and natural language processing.To capture the audio information as well as the temporal structure and semantic relationship between events,this paper proposes a dense video captioning method based on multi-modal features.Firstly,Timeception layer is used as basic module in action proposal generation stage to better adapt various time span of action segments.Secondly,audio features are used to enhance the effect of proposal and description generation stages.Finally,the temporal semantic relation module models the temporal structure and semantic information between events to further enhance the accuracy of description generation.In addition,this paper also constructs a dataset named SDVC to explore the effectiveness of this method in application of real learning scene.The experimental results on ActivityNet Captions and SDVC datasets show that the AUC of action proposal generation increases by 0.8%and 6.7%,respectively;and in turn,using generated action proposals for description generation,BLEU_3 and BLEU_4 of SDVC dataset increased by 2.3%and 2.2%,respectively.