基于全局与局部融合的相关性对齐跨模态检索

Correlation-Aligned Cross-Modal Retrieval Based on Global and Local Fusion

哈希算法因其低存储和高效率而被广泛应用于跨模式检索深度哈希能够很好地提取多模态数据的特征,近年来受到越来越多的关注。然而,大多数用于跨模态检索的深度哈希方法没有充分利用使用所有信息,也没有充分挖掘异构数据的相关性。提出一种基于全局和局部特征的深度跨模态哈希与相关性对齐方法在该方法中,分别为图像和文本模态设计两个深度神经网络来分别提取图像和文本的全局和局部的特征,并学习两个哈希函数同时,保留异构数据特征的模态间相似性,这可以利用语义相关性其次,对异源数据的分布进行对齐,以便更好地挖掘模态间相关性。实验结果表明,该方法优于最先进的方法。

Hash algorithms are widely used for cross-modal retrieval due to their low storage and high efficiency. Deep hashing can extract features of multi-modal data well and has received more and more attention in recent years. However, most deep hashing methods for cross-modal retrieval do not make full use of all information and do not fully exploit the relevance of heterogeneous data. In this paper, we propose a deep cross-modal hashing and correlation alignment method based on global and local features. In our method, we design two deep neural networks for image and text modalities to extract global and local features of the image and text, respectively, and learn two hash functions. At the same time, we preserve the inter-modal similarity of the heterogeneous data features, which can exploit semantic correlation. Secondly,the distributions of the heterogeneous data are aligned to better exploit the inter-modal correlations. The experimental results show that our method outperforms state-of-the-art methods.