基于多模态异质动态融合的情绪分析研究

Dynamic Fusion of Multi-modal Heterogeneous Data for Sentiment Analysis

近年来,利用多模态数据进行情绪分析是一个非常热门的领域。如何对模态内部信息及模态之间的相互作用进行更好的利用,是一个值得探讨的研究问题。而多个模态之间的相互作用,并不是一个静态的过程,而是动态变化的,且模态对于不同的任务而言也存在动态的强弱差异。若不能妥善处理,将导致模型性能的下降。该文针对时序多模态情绪数据提出了一种异质的动态融合方法,通过层次化的异质动态融合方式更完备地进行模态融合,并且动态地捕捉到模态间的相互作用。因此,该方法在提高模型性能的同时也提高了模态融合过程的可解释性。同时,该文利用多任务学习策略,将异质动态融合网络联合单个模态的自监督学习网络,获得模态的一致性及差异性特征。通过CMU-MOSI及CMU-MOSEI数据集上的实验表明该模型相比于主流模型具有优势,且模态融合的过程更具可解释性。

In recent years, sentiment analysis has been extended to multi-modal data, and the dynamic instead of static interaction of the intra modality data is worth exploring. This paper proposes a dynamic fusion method for heterogeneous multi-modal emotional stream data to completely capture the interaction between modalities. And using multi-task learning strategy, the heterogeneous dynamic fusion network is combined with a single modality self-supervised learning network to obtain the consistency and difference characteristics of the modality. Experiments on the CMU-MOSI and CMU-MOSEI indicate the advantage of the proposed method over mainstream models, as well as its interpretability.