摘要
针对现有基于注意力机制的多模态学习,对文字上下文之间的自我联系和图像目标区域的空间位置关系进行了深入研究。在分析现有注意力网络的基础上,提出使用自注意力模块(self-attention,SA)和空间推理注意力模块(spatial reasoning attention,SRA)对文本信息和图像目标进行映射,最终得到融合特征输出。相较于其他注意力机制,SA和SRA可以更好地将文本信息匹配图像目标区域。模型在VQAv2数据集上进行训练和验证,并在VQAv2数据集上达到了64.01%的准确率。
Aiming at the existing multi-modal learning which based on attention mechanism,this paper studied the self-association between the context of the text and the spatial positional relationship of the object area of the image.Based on the analysis of existing attention networks,this paper proposed to use SA and SRA to map the text information to the image object,and finally obtained the fusion feature output.Compared with other attention mechanisms,SA and SRA can better match text information to the image object area.The model is trained and verified on the VQAv2 dataset and achieves an accuracy of 64.01%on the VQAv2 dataset.
作者
李智涛
周之平
叶琴
Li Zhitao;Zhou Zhiping;Ye Qin(School of Information Engineering,Nanchang Hangkong University,Nanchang 330063,China)
出处
《计算机应用研究》
CSCD
北大核心
2021年第3期952-955,共4页
Application Research of Computers
基金
国家自然科学基金资助项目(71761028)。
关键词
视觉问答
注意力机制
多模态学习
自注意力
空间推理注意力
visual question answering(VQA)
attention mechanism
multimodal learning
self-attention
spatial reasoning attention