基于差异化和空间约束的自动图像描述模型

Image captioning model based on divergence-based and spatial consistency constraints

多头注意力机制是图像描述模型的常用方法,该机制通过多分支结构构建关于输入特征的独特属性,以提高特征模型的区分性。然而,不同分支的独立性导致建模存在冗余性。同时,注意力机制会关注于不重要的图像区域,导致描述的文本不够准确。针对上述问题,提出一种损失函数作为训练目标的正则化项,以提高多头注意力机制的多样性和准确性。在多样性方面,提出一种多头注意力的差异化正则,鼓励多头注意力机制的不同分支关注于所描述目标的不同部件,使不同分支的建模目标变得简单。同时,不同分支相互融合,最后形成完整且更有区分性的视觉描述。在准确性方面,设计一种空间一致性正则。通过建模多头注意力机制的空间关联,鼓励注意力机制关注的图像区域尽可能集中,从而抑制背景区域的影响,提高注意力机制的准确性。提出差异化正则和空间一致性正则共同作用的方法,最终提升自动图像描述模型的准确性。所提方法在MS COCO数据集上对模型进行验证,并与多种代表性工作进行对比。实验结果表明:所提方法显著地提高了图像描述的准确性。

The multi-head attention mechanism has been widely adopted in image captioning.It is appealing for the ability to jointly attend to information from different representation subspaces.However,as each head captures distinct properties of the input individually,the diversity between heads’representations is not guaranteed.In the meanwhile,most existing attention models encounter the problem of“attention defocus”,i.e.,they fail to concentrate on correct image regions when generating the target words.Consequently,the generated sentences are not accurate enough.To address these problems,we propose a novel training objective that serves as an auxiliary regularization function to improve the diversity and accuracy of the multi-head attention mechanism.In the beginning,we present a divergence-based regularization that encourages each brain to concentrate on various areas of the goal.Partial representations are aggregated to produce distinct representations of the target.Secondly,we introduce a spatial consistency regularization that builds the spatial relationship among the attended regions.By encouraging the attended regions to be focussed,it enhances image captioning.We proposed a method for the joint action of divergence-based regularization and spatial consistency regularization.We compare the performance of the proposed method with state-of-the-art methods on challenging MS COCO datasets.The experimental results demonstrate the superior performance of the proposed method.