基于双向注意力机制的图像描述生成

Image Captioning Based on Bidirectional Attention Mechanism

结合注意力机制的编码器—解码器框架被广泛应用于图像描述生成任务中。以往方法中,注意力机制根据当前时刻的语义信息挑选出重要的局部图像特征,进而依靠解码器的"翻译"能力将图像特征解码成文字。然而,在此过程中,单向的注意力机制并未检验语义信息与图像内容的一致性。因此,所生成的描述在准确性方面有所欠缺。为解决上述问题,该文提出一种基于双向注意力机制的图像描述生成方法,在单向注意力机制的基础上,加入图像特征到语义信息方向上的注意力计算,实现图像和语义信息两者在两个方向上的交互,并设计了一种门控网络对上述两个方向上的信息进行融合。最终,提高解码器所蕴含的语义信息与图像内容的一致性,使得所生成描述更加准确。此外,与前人研究不同的是,该文在注意力模块中利用了历史时刻的语义信息辅助当前时刻的单词生成,并对历史语义信息的作用进行了验证。该文基于MSCOCO和Flickr30k两种图像描述生成数据集,并使用两种图像特征进行了实验。实验结果显示,在MSCOCO数据集上,BLEU4分值平均提升1.3,CIDEr值平均提升6.3。在Flickr30k数据集上,BLEU4分值平均提升0.9,CIDEr值平均提升2.4。

The attention-based encoder-decoder framework is widely used in image captioning.In previous methods,the single-directional attention mechanism does not check the consistency between semantic information and image content,causing low accuracy in the generated caption.In order to solve the above problem,this paper proposes an image captioning method based on bi-directional attention mechanism.On the basis of the single-directional attention mechanism,the attention calculation is added from image feature to the semantic information,enabling the interaction between the image and the semantic information in two directions.This paper designs a gated network to fuse information in the above two directions.In contrast to previous studies,this paper uses the historical semantic information to assist in current word generation in the attention module.Using two types of image features,the experimental results show that on MSCOCO dataset,the BLEU4 score is increased by 1.3 and the CIDEr score by 6.3 in average.And on Flickr30 k,the BLEU4 score is increased by 0.9 and the CIDEr score by 2.4 in average.