融合语义特征和分布特征的跨媒体关联分析方法研究

Research on Cross-media Correlation Analysis by Fusing Semantic Features and Distribution Features

文本、图像、视频、音频等多种媒体数据具有多源异构的特性,这导致“语义鸿沟”问题的出现。现有文献采用的方法中大多数是针对文本和图像两种媒体数据展开研究,难以实现更多类型媒体数据的关联分析。因此,本文融入多种媒体数据的语义特征和分布特征,来对跨媒体关联分析方法进行深入研究,以实现文本、图像、视频、音频等多种媒体数据的一致性表示。首先,对多种媒体数据进行向量化表示,并输入模型;其次,利用双向长短期记忆网络(bidirectional long short-term memory,BiLSTM)挖掘输入数据的上下文信息,得到各种媒体数据的特征向量;最后,融合特征向量的语义特征和分布特征进行跨媒体关联分析,进而得到跨媒体的一致性表示。自建数据集上的比较实验结果表明,本文的研究方法较之CCA(canonical correlation analysis)、KCCA(kernel canonical correlation analysis)、Deep-SM(deep semantic match)等已有方法具有更高的关联分析准确率,这表明本文的研究方法能够较为准确地发现各种媒体数据之间的语义关联关系。希望本文的研究对跨媒体关联分析研究具有一定的指导和借鉴作用。

Several types of media data such as text,image,video,and audio are of multi-source heterogeneous type,which leads to the problem of semantic gaps.Current researches focus mostly on text and image,presumably because it is difficult to measure the correlation between more types of media data.Therefore,we discuss performing cross-media correlation analysis by fusing the semantic features and distribution features so as to produce consistent presentation of different types of media data.The different types of media data are first vectorized and input into the proposed model.Then,bidirectional long short-term memory(BiLSTM)is utilized to extract the context information,and the feature vectors are obtained.Finally,the correlation between different types of media data is analyzed by fusing the semantic features and distribution features,and all types of media data are represented consistently.The comparative experimental results show that the method proposed in this paper performs better than several traditional methods such as CCA(canonical correlation analysis),KCCA(kernel canonical correlation analysis),and Deep-SM(deep semantic match),which indicates that the proposed method can precisely detect the semantic correlation between different types of media data.The paper offers guidance and reference for research on cross-media correlation analysis.