基于跨模态Transformer的多模态细粒度情感分析方法研究

Research on Multimodal Fine-grained Sentiment Analysis Method Based on Cross-modal Transformer

多模态细粒度情感分析任务要求根据文本以及对应的图片信息,判断文本中每个观点实体的情感倾向。目前基于文本的细粒度情感分析模型无法对多模态数据进行建模,因此论文针对多模态细粒度情感分析任务,提出了一种基于跨模态Transformer的层次化神经网络模型。该模型利用从预训练模型中抽取出的文本以及图片特征,通过调整跨模态Transformer的输入来建模文本和图像之间的信息交互,从而达到利用图片信息来辅助文本信息进行情感分析的目的。论文在两个真实的多模态社交媒体数据集上开展了实验,并与其他方法展开对比,验证了论文提出的方法的有效性。

The multimodal fine-grained sentiment analysis task requires to predict the sentiment polarities of each opinion target based on the text and corresponding image information. In this paper,a hierarchical neural network model is proposed based on cross-modal Transformer for multimodal fine-grained sentiment analysis. This model adopts the text and image features extracted from the pre-trained model to model the interaction between text and image by adjusting the input of cross-modal Transformer and achieve the purpose of using the image information to assist text information for sentiment analysis. Experiments are carried out on two real multimodal social media data sets and compared with other methods. Experimental results verify the effectiveness of the proposed method.