基于语义增强特征融合的多模态图像检索模型

Multimodal image retrieval model based on semantic-enhanced feature fusion

为了在多模态图像检索任务中建立文本特征与图像特征的相关性,提出基于语义增强特征融合的多模态图像检索模型(SEFM).该模型通过文本语义增强模块、图像语义增强模块2部分在特征融合时对组合特征进行语义增强.在文本语义增强模块建立多模态双重注意力机制,利用双重注意力建立文本与图像之间的关联以增强文本语义;在图像语义增强模块引入保留强度和更新强度,控制组合特征中查询图像特征的保留和更新程度.基于以上2个模块可以优化组合特征使其更接近目标图像特征.在MIT-States和Fashion IQ这2个数据集上对该模型进行评估,实验结果表明在多模态图像检索任务上该模型与现有方法相比在召回率和准确率上都有所提升.

A multimodal image retrieval model based on semantic-enhanced feature fusion(SEFM) was proposed to establish the correlation between text features and image features in multimodal image retrieval tasks. Semantic enhancement was conducted on the combined features during feature fusion by two proposed modules including the text semantic enhancement module and the image semantic enhancement module. Firstly, to enhance the text semantics, a multimodal dual attention mechanism was established in the text semantic enhancement module, which associated the multimodal correlation between text and image. Secondly, to enhance the image semantics, the retain intensity and update intensity were introduced in the image semantic enhancement module, which controlled the retaining and updating degrees of the query image features in combined features. Based on the above two modules,the combined features can be optimized, and be closer to the target image features. In the experiment part, the SEFM model was evaluated on MIT-States and Fashion IQ datasets, and experimental results show that the proposed model performs better than the existing works on recall and precision metrics.