图像特征注意力与自适应注意力融合的图像内容中文描述

Chinese description of image content based on fusion of image feature attention and adaptive attention

针对现有基于注意力机制的图像内容中文描述模型无法在关注信息不减弱和无缺失的条件下对重点内容进行注意力加强关注的问题,提出一种图像特征注意力与自适应注意力融合的图像内容中文描述模型。模型使用编解码结构,首先在编码器网络中提取图像特征,并通过图像特征注意力提取图像全部特征区域的注意力信息;然后使用解码器网络将带有注意力权重的图像特征解码生成隐藏信息,以保证关注信息不减弱、无缺失;最后利用自适应注意力的视觉哨兵模块对图像特征中的重点内容进行再次加强关注,从而更加精准地提取图像的主体内容。使用多种评价指标(BLEU、METEOR、ROUGEL和CIDEr)进行模型验证,将所提模型与单一基于自适应注意力和基于图像特征注意力的图像描述模型进行对比实验,该模型的CIDEr评价指标值分别提高了10.1%和7.8%;同时与基线模型NIC(Neural Image Caption)以及基于自底向上和自顶向下(BUTD)注意力的图像描述模型相比,该模型的CIDEr评价指标值分别提高了10.9%和12.1%。实验结果表明,所提模型的图像理解能力得到了有效提升,其各项评价指标得分均优于对比模型。

Aiming at the problem that the existing Chinese description models of image content based on attention mechanism cannot focus on the key content without weakening or missing attention information,a Chinese description model of image content based on fusion of image feature attention and adaptive attention was proposed.An encode-decode structure was used in this model.Firstly,the image features were extracted in the encoder network,and the attention information of all feature regions of the image was extracted by the image feature attention.Then,the decoder network was used to decode the image features with attention weights to generate hidden information,so as to ensure that the attention information was not weakened or missed.Finally,the visual sentry module of self-adaptive attention was used to focus on the key content in the image features again,so that the main content of the image was able to be extracted more accurately.Several evaluation indices including BLEU,METEOR,ROUGEL and CIDEr were used to verify the models,the proposed model was compared with the image description models based on self-adaptive attention or image feature attention only,and the proposed model had the evaluation value of CIDEr improved by 10.1%and 7.8%respectively.Meanwhile,compared with the baseline model Neural Image Caption(NIC)and the Bottom-Up and Top-Down(BUTD)attention based image description model,the proposed model had the evaluation index value of CIDEr increased by 10.9%and 12.1%respectively.Experimental results show that the image understanding ability of the proposed model is effectively improved,and the score of each evaluation index of the model is better than those of the comparison models.