MSAM:针对视频问答的多阶段注意力模型

MSAM:Video Question Answering Based on Multi-Stage Attention Model

视频问答(VideoQA)任务需要理解视频和问题中的语义信息生成答案。目前,基于注意力模型的VideoQA方法很难完全理解和准确定位与问题相关的视频信息。为解决上述问题,提出一种基于注意力机制的多阶段注意力模型网络(MSAMN)。该网络将视频、音频以及文本等多模态特征输入到多阶段注意力模型(MSAM)中,通过逐阶段的定位方式精准找到与回答问题相关的视频信息,用于答案生成。为了提高特征融合的有效性,提出一种三模态压缩级联双线性(TCCB)算法计算不同模态特征之间的相关性。MASMN在ZJL数据集上进行实验,平均准确率均为54.3%,比传统方法提高了近15%,比现有方法提高了近7%。

The video question answering(VideoQA)task requires understanding of semantic information of both the video and question to generate the answer.At present,it is difficult for VideoQA methods that are based on attention model to fully understand and accurately locate video information related to the question.To solve this problem,a multi-stage attention model network(MSAMN)is proposed.This network extracts multi-modal features such as video,audio and text and feeds these features into the multi-stage attention model(MSAM),which is able to accurately locate the video information through a stage-by-stage localization method.In order to improve the effectiveness of feature fusion,a triple-modal compact concat bilinear(TCCB)algorithm is proposed to calculate the correlation between different modal features.This network is tested on the ZJL dataset.The average accuracy rate is 54.3%,which is nearly 15%higher than the traditional method and nearly 7%higher than the exist method.