基于多模态融合和时空上下文语义的跨媒体检索模型的研究

Research of cross-media information retrieval model based on multimodal fusion and temporal-spatial context semantic

如何跨越低层特征描述到高层语义知识的"语义鸿沟"已成为跨媒体检索(CMR)问题的关键,提出一个基于多模态融合描述和时空上下文语义的跨媒体检索模型,对多模态融合的特征采用主成分分析(PCA)和独立成分分析(ICA)相结合的降维算法、采用基于支持向量机(SVM)和隐马尔可夫模型(HMM)的混合分类器进行语义映射,同时给出了时空模糊聚类分析方法和基于相关反馈的跨媒体检索算法。并在此基础上开发出基于该模型的原型系统,成功验证了该模型的可行性和正确性,可为相关系统的设计者提供思路。

The solution of ＂ semantic gap＂ between the low-level features describing and the high-level semantic knowledge has become the key in problems of the Cross-Media Retrieval （CMR）, a CMR model based on multimodal fusion and temporal-spatial context semantic was designed. The Independent Component Analysis （ICA） and Principal Component Analysis （PCA） were applied to dimension reduction of multimodal fusion features. The classifier of Support Veetor Machine （SVM） and Hidden Markov Model （HMM） was designed to map semantic relationship in the model; meanwhile, methods of temporal-spatial fuzzy cluster and relevance feedback were used to improve the effect of CMR system. A prototype based on the model had been developed, and validated the correctness of the new model, which can provide enlightenment to the designers who work at CMR system.