融合动态梯度和多视图协同注意力的情感分析

Sentiment Analysis Combining Dynamic Gradient and Multi-view Co-attention

针对多模态情感分析中模态间优化不平衡和多模态特征融合不充分的问题,提出一种融合动态梯度机制和多视图协同注意力机制的多模态情感分析模型(DG-MCM),能够有效挖掘单模态特征并充分融合多模态信息。首先,模型使用预训练模型BERT和堆叠式长短期记忆神经网络(SLSTM)学习文本、音频和视频的特征,并提出动态梯度机制,通过监测各模态对学习目标的贡献差异和学习速度辅助各模态的特征学习。其次,将得到的不同模态的特征使用多视图协同注意力机制进行特征融合,通过将每两个模态投影到多个空间执行交互获得更加充分的融合特征。最后,拼接融合特征和单模态特征进行情感预测。在CMU-MOSI和CMUMOSEI数据集的实验结果表明,该模型能够充分学习单模态和不同模态之间的信息,有效提升多模态情感分析的准确率。

Aiming at the problems of unbalanced inter-modal optimization and inadequate fusion of multimodal features in multimodal sentiment analysis,a multimodal sentiment analysis model combining dynamic gradient mechanism and multi-view co-attention mechanism(DG-MCM)is proposed,which can effectively mine single-modal representation and fully integrate multimodal information.Firstly,the model uses pre-trained model BERT(bidirectional encoder representation from transformers)and stacked long short-term memory(SLSTM)to learn the features of text,audio and video,and proposes a dynamic gradient mechanism.By monitoring the contribution difference and learning speed of each mode,the feature learning of each mode is assisted.Secondly,the features of different modes obtained are fused using the multi-view co-attention mechanism.By projecting every two modes into multiple spaces for interaction,more adequate fusion features are obtained.Finally,fusion features and single-modal features are spliced together for sentiment prediction.Experimental results on CMU-MOSI and CMU-MOSEI datasets show that this model can fully learn information between single mode and different modes,and effectively improve the accuracy of multimodal sentiment analysis.