基于类时序注意力机制的图像描述方法

Image Captioning with Similar Temporal Attention Mechanism

近年来,注意力机制已经广泛应用于计算机视觉领域,图像描述常用的编码器-解码器框架也不例外.然而,当前的解码框架并未较清楚地分析图像特征与长短期记忆神经网络(LSTM)隐藏状态之间的相关性,这也是引起累积误差的原因之一.基于该问题,本文提出一个类时序注意力网络(Similar Temporal Attention Network,STAN),该网络扩展了传统的注意力机制,目的是加强注意力结果与隐藏状态在不同时刻的相关性.STAN首先对当前时刻的隐藏状态和特征向量施加注意力,然后通过注意力融合槽(AFS)将两个相邻LSTM片段的注意力结果引入到下一时刻的网络循环中,以增强注意力结果与隐藏状态之间的相关性.同时,本文设计一个隐藏状态开关(HSS)来指导单词的生成,将其与AFS结合起来可以在一定程度上解决累积误差的问题.在官方数据集Microsoft COCO上的大量实验和各种评估机制的结果表明,本文提出的模型与基线模型相比,具有明显的优越性,取得了更有竞争力的结果.

Recently,attention mechanisms have been widely used in computer vision in such aspects as the common encoder/decoder framework for image captioning.However,the current decoding framework does not clearly analyze the correlation between image features and the hidden states of the Long Short-Term Memory(LSTM)network,leading to cumulative errors.In this study,we propose a Similar Temporal Attention Network(STAN)that extends conventional attention mechanisms to strengthen the correlation between attention results and hidden states at different moments.STAN first applies attention to the hidden state and feature vector at the current moment,and then introduces the attention result of two adjacent LSTM segments into the recurrent LSTM network at the next moment through an Attention Fusion Slot(AFS)to enhance the correlation between attention results and hidden states.Also,we design a Hidden State Switch(HSS)to guide the generation of words,which is combined with the AFS to reduce cumulative errors.According to the extensive experiments on the public benchmark dataset Microsoft COCO and various evaluation mechanisms,our algorithm is superior to the baseline model and can get more competitive attention results.