摘要
为了在多模态图像检索任务中建立文本特征与图像特征的相关性,提出基于语义增强特征融合的多模态图像检索模型(SEFM).该模型通过文本语义增强模块、图像语义增强模块2部分在特征融合时对组合特征进行语义增强.在文本语义增强模块建立多模态双重注意力机制,利用双重注意力建立文本与图像之间的关联以增强文本语义;在图像语义增强模块引入保留强度和更新强度,控制组合特征中查询图像特征的保留和更新程度.基于以上2个模块可以优化组合特征使其更接近目标图像特征.在MIT-States和Fashion IQ这2个数据集上对该模型进行评估,实验结果表明在多模态图像检索任务上该模型与现有方法相比在召回率和准确率上都有所提升.
A multimodal image retrieval model based on semantic-enhanced feature fusion(SEFM) was proposed to establish the correlation between text features and image features in multimodal image retrieval tasks. Semantic enhancement was conducted on the combined features during feature fusion by two proposed modules including the text semantic enhancement module and the image semantic enhancement module. Firstly, to enhance the text semantics, a multimodal dual attention mechanism was established in the text semantic enhancement module, which associated the multimodal correlation between text and image. Secondly, to enhance the image semantics, the retain intensity and update intensity were introduced in the image semantic enhancement module, which controlled the retaining and updating degrees of the query image features in combined features. Based on the above two modules,the combined features can be optimized, and be closer to the target image features. In the experiment part, the SEFM model was evaluated on MIT-States and Fashion IQ datasets, and experimental results show that the proposed model performs better than the existing works on recall and precision metrics.
作者
杨帆
宁博
李怀清
周新
李冠宇
YANG Fan;NING Bo;LI Huai-qing;ZHOU Xin;LI Guan-yu(School of Information Science and Technology,Dalian Maritime University,Dalian 116026,China)
出处
《浙江大学学报(工学版)》
EI
CAS
CSCD
北大核心
2023年第2期252-258,391,共8页
Journal of Zhejiang University:Engineering Science
基金
国家自然科学基金资助项目(61976032,62002039)
辽宁省教育厅科学研究面上资助项目(LJKZ0063)。
关键词
多模态
语义增强
特征融合
图像检索
注意力机制
multimodality
semantic enhancement
feature fusion
image retrieval
attention mechanism