基于ViLT的社交媒体领域图文情感分析方法

Image-Text Sentiment Analysis Method in Social Media Domain Based on ViLT

现有的图文情感分析方法更多地集中于图文信息的特征提取方面,较少关注不同模态之间的特征对齐,针对这一问题提出了一种基于ViLT (Vision-and-Language Transformer)的社交媒体领域图文情感分析方法。结合社交媒体文本长度较短、语法不规范等特点,选用BERTweet作为文本编码器,利用ViLT模型将图片切片投影的方法提取图像特征。将文本特征与图像特征进行拼接,送入同一个Transformer模块,得到基于图文多模态分析的情感结果。并充分挖掘文本与图像自身的特征得出两个基于单模态的情感分析结果,最后对三种情感分析结果使用加权融合策略确定最终的情感极性。该方法在公开数据集上进行了实验,验证了本文情感分类方法的有效性。

The existing image-text sentiment analysis methods focus more on feature extraction of image and text information with less attention to feature alignment between different modalities. Therefore, this paper proposes an image-text sentiment analysis method in the social media domain based on Vision-and-Language Transformer (ViLT). Combining the features of short length and irregular syntax of social media texts, BERTweet is chosen as the text encoder and image features are extracted by slicing and projecting images using ViLT model. The text features and image features are stitched together and sent to the same Transformer module to get the sentiment results based on the multimodal analysis of the graphical text. And the features of text and image themselves are fully exploited to derive two unimodal-based sentiment analysis results. Finally, the final sentiment polarity is determined using a weighted fusion strategy for the three sentiment analysis results. The method is experimented on a public dataset to verify the effectiveness of the sentiment classification method in this dissertation.