基于跨模态融合ERNIE的多模态情感分析研究

Multi-modal Sentiment Analysis based on Cross-modal Fusion ERNIE

针对情感分析主要集中于单模态文本数据,忽略多模态数据融合问题,通过结合屏蔽多模态注意力方式,提出跨模态融合ERNIE的情感分析模型(CM-ERNIE)。首先,使用CNN和BiGRU提取音频数据特征以及词向量提取文本序列特征;其次,通过屏蔽多模态注意力作为CM-ERNIE的核心单元动态调整文本和音频数据权重,最后,文本和音频模态的交互作用微调预训练ERNIE模型。该模型在多模态电影评论观点数据集CMU-MOSEI和CMU-MOSI上评估。实验表明,模型在多模态数据集CMU-MOSEI和CMU-MOSI上评估该模型比单模态情感分析模型准确度高,并且多模态情感分析的研究蕴含巨大的价值,可为多模态场景下的情感分析、舆情分析和意图识别等实际应用问题提供决策支持。

Aiming at the fact that sentiment analysis mainly focuses on single-modal text data and ignores the problem of multi-modal data fusion,a cross-modal fusion ERNIE sentiment analysis model(CM-ERNIE)is proposed by combining the masked multi-modal attention method.First,use CNN and BiGRU to extract audio data features and word vectors to extract text sequence features;second,dynamically adjust text and audio data weights by masking multimodal attention as the core unit of CM-ERNIE,and finally,text and audio modalities The interaction of fine-tuning the pretrained ERNIE model.The model is evaluated on the multimodal movie review opinion datasets CMU-MOSEI and CMU-MOSI.Comprehensive experiments show that the model is more accurate than the single-modal sentiment analysis model on the multi-modal datasets CMU-MOSEI and CMU-MOSI,and the research of multi-modal sentiment analysis contains great value,which can be used for multi-modal sentiment analysis.It provides decision support for practical application problems such as sentiment analysis,public opinion analysis,and intent recognition in modal scenarios.