融合多特征和注意力机制的多模态情感分析模型

Multimodal Sentiment Analysis Model Integrating Multi-features and Attention Mechanism

【目的】针对当前多模态情感分析中多模态特征提取不充分,模态内部信息和模态间交互信息结合不充分的问题,提出一种融合多特征和注意力机制的多模态情感分析模型。【方法】在多模态特征提取方面,增加视频模态中人物的肢体动作、性别和年龄特征;对于文本模态,融合基于BERT的字粒度语义向量和融合义原信息的词粒度语义向量,丰富了多模态数据的低层特征。利用自注意力机制和跨模态注意力机制以实现模态内部信息和模态间信息的充分结合。将各模态特征进行拼接,通过软注意力机制为各模态特征分配注意力权重,通过全连接层输出最终的情感分类结果。【结果】在公开数据集CH-SIMS和本文构建的热点舆情评论视频数据集HPOC上与Self-MM模型对比,实验结果表明,本文模型在CH-SIMS数据集上的二分类准确率、三分类准确率和F1值分别提升1.83、1.74和0.69个百分点,在HPOC数据集上分别提升1.03、0.94和0.79个百分点。【局限】视频中人物所处的场景可能不断变化,不同的场景可能蕴含不同的情感信息,模型未考虑融合人物所处的场景信息。【结论】本文模型丰富了多模态数据的低层特征,充分结合模态内部信息和模态间信息,能够有效提升情感分析的效果。

[Objective]This paper proposes a multimodal sentiment analysis model integrating multiple features and attention mechanisms.It addresses the insufficient extraction of multimodal features and inadequate interaction of intra-modal and inter-modal information in existing models.[Methods]In multimodal feature extraction,we enhanced the features of body movements,gender,and age of individuals in the video modality.For the text modality,we integrated BERT-based character-level and word-level semantic vectors.Therefore,we enriched the low-level features of multimodal data.We also utilized self-attention and cross-modal attention mechanisms to integrate intra-modal and inter-modal information.We concatenated the modal features and employed a soft attention mechanism to allocate attention weight to each feature.Finally,we generated the sentiment classification results through fully connected layers.[Results]We examined the proposed model on the public dataset(CH-SIMS)and the Hot Public Opinion Comments Videos(HPOC)dataset constructed in this paper.Compared with the Self-MM model,our model improved the binary classification accuracy,tri-class classification accuracy,and F1 value by 1.83%,1.74%,and 0.69% on the CH-SIMS dataset,and 1.03%,0.94%,and 0.79% on the HPOC dataset.[Limitations]The person’s scene in the video may change constantly,and different scenes may contain different emotional information.Our model does not integrate the scene information of the person.[Conclusions]The proposed model enriches the low-level features of multimodal data and improves the effectiveness of sentimental analysis.