基于感知融合的多任务多模态情感分析模型

Multi-task&Multi-modal Sentiment Analysis Model Based on Aware Fusion

【目的】针对现有的多模态情感分析模型中存在的上下文信息利用不足、未注重模态间的一致性和差异性等问题,设计一种基于感知融合的多任务多模态情感分析模型。【方法】设计了多模态、文本、声音、图像情感分析4个任务,使用BERT、wav2vec 2.0、OpenFace 2.0模型提取文本、声音、图像的特征信息,经SelfAttention层处理后传入感知融合层进行多模态特征融合,最后使用Softmax对单模态信息和多模态信息进行分类。引入同方差不确定性损失函数,为不同任务自动分配权重。【结果】所提模型与基线模型相比,在CHSIMS数据集上准确率和F1值上分别提升1.59和1.67个百分点,在CMU-MOSI数据集上准确率和F1值上分别提升0.55和0.67个百分点。消融实验表明采用多任务学习比未采用多任务学习在准确率和F1值上分别提升4.08和4.18个百分点。【局限】未测试模型在大规模数据集上的表现。【结论】所提模型能够有效地降低噪声,提升多模态融合效果,同时多任务学习框架能够使模型获得更好的学习效果。

[Objective]This paper develops a multi-task and multi-modal sentiment analysis model based on aware fusion,aiming to sufficiently use context information,as well as address the modality-invariant and modality-specific issues.[Methods]We established multi-modal,text,acoustic,and image sentiment analysis tasks.We extracted their features using BERT,wav2vec2.0,and openface2.0 models,which were processed by the self-attention layer and sent to the aware fusion layer for multi-modal feature fusion.Finally,we categorized the single-modal and multi-modal information using Softmax.We also introduced the loss function of the homoscedastic uncertainty to assign weights to different tasks automatically.[Results]Compared with the baseline method,the proposed model improved the accuracy and F1 value by 1.59%and 1.67%on CH-SIMS,and 0.55%and 0.67%on CMU-MOSI.The ablation experiment showed that the accuracy and F1 value of multi-task learning were 4.08%and 4.18%higher than those of single-task learning.[Limitations]We need to examine the new model’s performance on large-scale data sets.[Conclusions]The model can effectively reduce noise and improve multi-modal fusion.The multi-task learning framework could also achieve better performance.