基于预训练表示学习的端到端跨媒体检索方法

An End-to-End Cross-Media Retrieval Method Based on Pre-Trained Representation Learning

跨媒体检索是集成媒体数据表示学习与媒体数据信息对齐的检索方式,现有的跨媒体表示学习方法没有将单一媒体数据表示学习的先进方法集成应用,在跨媒体信息高层表示方面缺乏有效语义对齐。提出一种预训练表示学习的端到端跨媒体检索方法,该方法采用先进的预训练表示学习方法,分别利用残差网络(ResNet)和BERT模型抽取图像和文本高层表示特征,然后利用自注意力机制挖掘跨媒体数据的语义关联,实现跨媒体信息的语义对齐。以平均精度均值作为评价指标,在3个广泛使用的跨媒体数据集上验证了模型的有效性。实验表明,所提方法在3个数据集上的平均精度均值都优于其他几种对比方法。

Cross-media retrieval is a retrieval method of integrating media data representation learning and media data information alignment. The existing cross-media representation learning method does not integrate the state-of-the-art methods of single media data representation learning and lacks effective semantic alignment in the aspect of high level representation of transmedia information. In this paper, an end-to-end cross-media retrieval method based on pre-trained representation learning is proposed. This method adopts advanced representation learning method, uses residual network(ResNet) to extract image features and bidirectional encoder representation from transfromers(BERT) model to extract text features respectively. Self-attention mechanism is used to mine semantic associations of cross-media data to achieve semantic alignment of cross-media information. We take the mean average precision as the evaluation metric and verify the validity of the model on three widely used cross-media datasets. Experimental results show that the average accuracy of the proposed method is superior to other methods in all three data sets.