基于自适应门控信息融合的多模态情感分析

Multimodal Sentiment Analysis Based on Adaptive Gated Information Fusion

多模态情感分析的目标是使用由多种模态提供的互补信息来实现可靠和稳健的情感分析。近年来,通过神经网络提取深层语义特征,在多模态情感分析任务中取得了显著的效果。而多模态信息的不同层次的特征融合也是决定情感分析效果的重要环节。因此,提出了一种基于自适应门控信息融合的多模态情感分析模型(AGIF)。首先,通过门控信息融合网络将Swin Transformer和ResNet提取的不同层次的视觉和色彩特征根据对情感分析的贡献进行有机融合。其次,由于情感的抽象性和复杂性,图像的情感往往由多个细微的局部区域体现,而迭代注意可以根据过去的信息精准定位这些情感判别区域。针对Word2Vec和GloVe无法解决一词多义的问题,采用了最新的ERNIE预训练模型。最后,利用自动融合网络“动态”融合各模态特征,解决了(拼接或TFN)确定性操作构建多模态联合表示所带来的信息冗余问题。在3个公开的真实数据集上进行了大量实验,证明了该模型的有效性。

The goal of multimodal sentiment analysis is to achieve reliable and robust sentiment analysis by utilizing complementary information provided by multiple modalities.Recently,extracting deep semantic features by neural networks has achieved remarkable results in multimodal sentiment analysis.But the fusion of features at different levels of multimodal information is also an important part in determining the effectiveness of sentiment analysis.Thus,a multimodal sentiment analysis model based on adaptive gating information fusion(AGIF)is proposed.Firstly,the different levels of visual and color features extracted by swin transformer and ResNet are organically fused through a gated information fusion network based on their contribution to sentiment analysis.Secondly,the sentiment of an image is often expressed by multiple subtle local regions due to the abstraction and complexity of sentiment,and these sentiment discriminating regions can be located accurately by iterative attention based on past information.The latest ERNIE pre-training model is utilized to solve the problem of Word2Vec and GloVe’s inability to handle the word polysemy.Finally,the auto-fusion network is utilized to“dynamically”fuse the features of each modality,solving the pro-blem of information redundancy caused by the deterministic operation(concatenation or TFN)to construct multimodal joint representation.Extensive experiments on three publicly available real datasets demonstrate the effectiveness of the proposed model.