融合两级相似度的跨媒体图像文本检索

Cross-Media Image-Text Retrieval with Two Level Similarity

为了更好地揭示图像和文本之间潜在的语义关联,提出了一种融合两级相似度的跨媒体检索方法,构建两个子网分别处理全局特征和局部特征,以获取图像和文本之间更好的语义匹配.图像分为整幅图像和一些图像区域两种表示,文本也分为整个语句和一些单词两种表示.设计一个两级对齐方法分别匹配图像和文本的全局和局部表示,并融合两种相似度学习跨媒体的完整表示.在MSCOCO和Flickr30K数据集上的实验结果表明,本文方法能够使图像和文本的语义匹配更准确,优于许多当前先进的跨媒体检索方法.

To better reveal the latent semantic correlation between image and text,this paper proposes a cross media retrieval method by fusing two level similarity,which constructs two subnets to deal with global features and local features respectively so as to obtain better semantic matching between image and text.The image representation is divided into whole image and some image regions,and the text representation is also divided into whole sentence and some words.A two level alignment method is designed to match the global and local representation of image and text,and the two similarities are fused to learn the complete cross-media representation.The experimental results on MSCOCO and Flickr30K datasets show that the proposed method can make the semantic matching of image and text more accurate,and is superior to many state-of-the-art cross-media retrieval methods.