用于多模态语义分析的嵌入共识自动编码器

Embedding Consensus Autoencoder for Cross-modal Semantic Analysis

跨模态检索技术是一项近年来的研究热点。多模态数据具有异质性,而不同形式的信息之间又有着相似性。传统的单模态方法只能以一种方式重构原始数据,并未考虑到不同数据之间的语义相似性,不能进行有效的检索。因此,文中建立了一个跨模态嵌入共识自动编码器(Cross-Modal Semantic Autoencoder with Embedding Consensus,ECA-CMSA),将原始数据映射到低维共识空间以保留语义信息,学习出对应的语义代码向量,并引入参数来实现去噪。然后,考虑到各模态之间的相似性,采用自动编码器将特征投影关联到语义代码向量。此外,对低维矩阵进行正则化稀疏约束,以平衡重构误差。在4个多模态数据集上验证所提方法的性能,实验结果证明其查询结果有所提升,实现了有效的跨模态检索。进一步,ECA-CMSA还可以应用于与计算机和网络有关的领域,如深度学习和子空间学习。该模型突破了传统方法中的障碍,创新地使用深度学习方法将多模态数据转换为抽象的表达,使其可以获得更好的准确度和识别结果。

Cross-modal retrieval has become a topic of popularity,since multi-data is heterogeneous and the similarities between different forms of information are worthy of attention.Traditional single-modal methods reconstruct the original information and lacks of considering the semantic similarity between different data.In this work,an Embedding Consensus Autoencoder for Cross-Modal Semantic Analysis is proposed,which maps the original data to a low-dimensional shared space to retain semantic information.Considering the similarity between the modalities,an automatic encoder is utilized to associate the feature projection to the semantic code vector.In addition,regularization and sparse constraints are applied to low-dimensional matrices to balance reconstruction errors.The high dimentional data is transformed into semantic code vector.Different models are constrained by parameters to achieve denoising.The experiments on four multi-modal data sets show that the query results are improved and effective cross-modal retrieval is achieved.Further,ECA-CMSA can also be applied to fields related to computer and network such as deep and subspace learning.The model breaks through the obstacles in traditional methods,and uses deep learning methods innovatively to convert multi modal data into abstract expression,which can get better accuracy and achieve better results in recognition.