特征融合的装修案例跨模态检索方法

A cross-modal retrieval algorithm of decoration cases on feature fusion

目前家装客服系统中主要依靠人工方式进行装修案例检索,导致该系统不能满足用户对咨询服务快捷、及时的需求而且人力成本高,故提出一种基于特征融合的装修案例跨模态检索算法。针对多模态数据的语义信息挖掘不充分,模型检索精度低等问题,对现有的风格聚合模块进行改进,在原始模块中引入通道注意力机制,以此来为每组装修案例中不同图片的特征向量添加合适的权重,从而增强包含更多有用信息的重要特征并削弱其他不重要的特征。同时,为充分利用多模态信息,设计一种适用于检索场景下的多模态特征融合模块,该模块能够自适应地控制2种不同模态的特征向量进行一系列的融合操作,以实现跨模态数据间的知识流动与共享,从而生成语义更丰富、表达能力更强的特征向量,进一步提升模型的检索性能。在自建的装修案例多模态数据集上将该方法与其他方法进行比较,试验结果表明本文方法在装修案例检索上具有更优越的性能。

Currently,home decoration customer service systems chiefly depend on manual decoration case retrieval,which leads to the system not meeting user demand for fast and timely consulting service and high labor costs.Thus,we propose a feature fusion-based cross-modal retrieval algorithm for decoration cases.Aiming at the problems of insufficient semantic information mining of multimodal data and low accuracy of model retrieval,the existing style aggregation module is improved.The channel attention mechanism is introduced into the original module to add suitable weights to the feature vectors of different pictures in each group of decoration cases,thereby improving the important features that include more helpful information and weakening other unimportant features.Conversely,a multimodal feature fusion module is developed for retrieval scenarios to make full use of multimodal information.The module can adaptively control a series of fusion operations of feature vectors from two different modalities to achieve knowledge flow and sharing between cross-modal data,thereby producing feature vectors with richer semantics and stronger expressive ability to improve the retrieval performance of the model further.Our developed algorithm is compared with other methods on a self-built multimodal dataset of decoration cases,and results show that the algorithm performs better in decoration case retrieval.