基于多层语义融合的图文信息情感分类研究

Sentiment Classification of Image-Text Information with Multi-Layer Semantic Fusion

【目的】对海量不同模态的社交媒体数据进行有效的情感分析,更好地了解公众的情感和意见倾向。【方法】为充分挖掘图文之间的关联性和互补性,提出一种基于多层语义融合的社交媒体图文信息情感分类模型,首先通过文本-图像语义关联模型、图像-文本语义关联模型、多模态语义深度关联融合模型三个子模型挖掘图文之间的双向多层次语义关联,进而使用加权策略对三个子模型的情感分类得分进行决策级融合得到最终情感分类结果。【结果】在真实图文数据集上的实验结果表明,与最优基线模型相比,所提模型在各项评估指标均能达到最优,其中准确率提高了1.0百分点,F1值提高了1.2百分点。【局限】实验仅在一个数据集上进行,没有对模型的鲁棒性和可扩展性做进一步验证。【结论】所提模型在情感分类任务上能够更加充分地挖掘社交媒体图文信息之间的关联性和互补性。

[Objective]This paper conducts sentiment analysis of images and text on social media data,aiming to better understand the public’s emotions and opinion tendencies.[Methods]To fully explore the correlation and complementarity between images and text,this paper proposes an image-text sentiment classification model in social media based on multi-layer semantic fusion.There are three sub-models in our study:text-image semantic association model,image-text semantic association model,and multimodal semantic deep association fusion model.We used these sub-models to explore the bidirectional and multi-level semantic associations between images and text.Then,we obtained the final classification results using a weighting strategy on the sentiment classification scores generated by the three sub-models.[Results]We examined our model with real image-text data sets and found it achieved the best performance in all evaluation metrics.The accuracy and F1 values of our model were 1.0%and 1.2%better than those of the optimal baseline model.[Limitations]We only evaluated the model’s performance with one single dataset.More research is needed to examine the robustness and scalability of the model.[Conclusions]In the sentiment classification task,the proposed model could more effectively explore the correlation and complementarity between image and text information on social media.