基于跨模态交叉注意力网络的多模态情感分析方法

Multimodal Sentiment Analysis Based on Cross-Modal Cross-Attention Network

挖掘不同模态内信息和模态间信息有助于提升多模态情感分析的性能,本文为此提出一种基于跨模态交叉注意力网络的多模态情感分析方法。首先,利用VGG-16网络将多模态数据映射到全局特征空间;同时,利用Swin Transformer网络将多模态数据映射到局部特征空间;其次,构造模态内自注意力和模态间交叉注意力特征;然后,设计一种跨模态交叉注意力融合模块实现不同模态内和模态间特征的深度融合,提升多模态特征表达的可靠性;最后,通过Softmax获得最终预测结果。在2个开源数据集CMU-MOSI和CMU-MSOEI上进行测试,本文模型在七分类任务上获得45.9%和54.1%的准确率,相比当前MCGMF模型,提升了0.66%和2.46%,综合性能提升显著。

Exploiting intra-modal and inter-modal information is helpful for improving the performance of multimodal sen-timent analysis.So,a multimodal sentiment analysis based on cross-modal cross-attention network is proposed.Firstly,VGG-16 network is utilized to map the multimodal data into the global feature space.Simultaneously,the Swin Transformer network is used to map the multimodal data into the local feature space.And the intra-modal self-attention and inter-modal cross-attention features are constructed.Then,a cross-modal cross-attention fusion module is designed to achieve the deep fusion of the intra-modal and inter-modal features,enhancing the represen-tation reliability of the multimodal feature.Finally,the softmax function is used to obtain the results of the sentiment analysis.The experimental results on two open source datasets CMU-MOSI and CMU-MSOEI show that the proposed model can achieve an accuracy of 45.9%and 54.1%respectively in the seven-classification task.Compared with the current classical MCGMF model,the accuracy of the proposed model has improved by 0.66%and 2.46%,and the overall performance improvement is significant.