基于文本指导的层级自适应融合的多模态情感分析

Multimodal sentiment analysis based on text-guided hierarchical adaptive fusion

提出了一种基于文本模态指导的多模态层级自适应融合方法,以文本模态信息为指导实现多模态信息的层级自适应筛选及融合。首先,基于跨模态注意力机制实现两两模态之间的重要性信息表征;然后通过多模态自适应门控机制实现基于多模态重要信息的层级自适应融合;最后综合多模态特征和模态重要性信息实现多模态情感分析。在公共数据集MOSI和MOSEI上的实验结果表明:对比基线模型,本文所提方法在准确率与F_(1)值方面分别提升了0.76%和0.7%。

The paper proposes a multi-modal hierarchical fusion method based on text modal guidance,which uses text modal information as the guidance to achieve hierarchical adaptive screening and fusion of multi-modal information.Firstly,the importance information representation between two modalities is realized based on the cross-modal attention mechanism,then the hierarchical adaptive fusion based on the multimodal important information is realized through the multimodal adaptive gating mechanism,and finally the multimodal features are synthesized.and modal importance information to implement multimodal sentiment analysis.The experimental results on the public datasets MOSI and MOSEI show that the accuracy and F_(1)value of the baseline model have increased by 0.76% and 0.7%,respectively.