基于空间注意力推理机制的视觉问答算法研究

Algorithm of visual question answering based on spatial attention reasoning mechanism

针对现有基于注意力机制的多模态学习,对文字上下文之间的自我联系和图像目标区域的空间位置关系进行了深入研究。在分析现有注意力网络的基础上,提出使用自注意力模块(self-attention,SA)和空间推理注意力模块(spatial reasoning attention,SRA)对文本信息和图像目标进行映射,最终得到融合特征输出。相较于其他注意力机制,SA和SRA可以更好地将文本信息匹配图像目标区域。模型在VQAv2数据集上进行训练和验证,并在VQAv2数据集上达到了64.01%的准确率。

Aiming at the existing multi-modal learning which based on attention mechanism,this paper studied the self-association between the context of the text and the spatial positional relationship of the object area of the image.Based on the analysis of existing attention networks,this paper proposed to use SA and SRA to map the text information to the image object,and finally obtained the fusion feature output.Compared with other attention mechanisms,SA and SRA can better match text information to the image object area.The model is trained and verified on the VQAv2 dataset and achieves an accuracy of 64.01%on the VQAv2 dataset.