基于语感一致性的社交媒体图文情感分析

Sentiment analysis of social media images-text based on semantic sense consistency

针对现有的图文情感分析方法未能充分考虑图像和文本之间存在的语义不一致问题,以及未对图像和文本表达不同情感的数据做相应处理,从而导致分类不精准的现象,提出基于语感一致性的社交媒体图文情感分析(social media image-text sentiment analysis based on semantic sense consistency,SA-SSC)方法。首先,使用RoBERTa和双向门控循环单元(bidirectional gated recurrent unit,BiGRU)提取文本语义特征,使用ResNet101获取图像视觉特征;然后,采用指导注意力(guided attention,GA)从图像区域情感和文本内容找到表达用户情感的显著性区域,得到新的图像视觉特征;最后,利用协同注意力将2种模态的特征进行融合,进而完成情感分类。在本文构建的MMSD-CN中文社交媒体图文情感数据集和CCIR-YQ数据集上进行了实验验证,结果表明,SA-SSC方法可以有效减弱图文语感不一致对社交媒体图文情感分析造成的影响,在各项评价指标上均取得了较高的提升。

The existing image-text sentiment analysis methods fail to fully consider the semantic inconsistency between images and texts,and do not deal with data that express different emotions in images and texts,resulting in inaccurate classification.A social media image-text sentiment analysis method was proposed based on semantic sense consistency(SA-SSC)method.Firstly,RoBERTa and bidirectional gated recurrent unit(BiGRU)were adopted to extract text semantic features,and ResNet101 was used to obtain image visual features.Then guided attention(GA)was used to find the significant area expressing users’emotion from image area emotion and text context,so as to obtain new image visual features.Finally,collaborative attention was used to combine the two modalities,and the features were fused to complete sentiment classification.The experiment verification was carried out on the MMSD-CN Chinese social media image-text sentiment dataset constructed in this paper and CCIR-YQ dataset.The results show that the SA-SSC method can effectively reduce the impact of inconsistent semantic sense on social media image-text sentiment analysis,and have achieved high improvement.