融合交叉注意力的突发事件多模态中文反讽识别模型

A multimodal Chinese sarcasm detection model for emergencies based on cross attention

网民在社交媒体参与突发事件讨论时,时常会采用反讽修辞方式表达对事件的看法,此举导致情感分析的难度增加,且已有中文反讽识别对社交媒体中网民发布的多模态评论研究较少,有必要对图文多模态中文反讽识别进行深入研究。运用交叉注意力机制捕捉模态间的不一致性表达,提出融合交叉注意力的多模态中文反讽识别模型(fuse cross attention model,FCAM)。在模型中,首先运用TextCNN(text convolutional neural networks)和ResNet(deep residual network)分别提取中文文本浅层特征和图像特征,再运用交叉注意力机制分别得到文本层和图像层的注意力特征,按照残差方式分别实现文本浅层特征和文本层注意力特征的连接、图像特征和图像层注意力特征的连接,使用注意力机制融合2个特征表示,经过分类层得到反讽分类结果。基于某一地区新冠疫情期间相关话题的微博评论数据,构建出突发公共卫生事件多模态中文反讽数据集,在该数据集上试验验证,相较于基准模型,FCAM具有一定的优越性。

Internet users often use sarcasm when discussing emergencies on social media,which complicates emotional analysis.In addition,there is a lack of research on multimodal comments,particularly those in Chinese,and their use of sarcasm on social media platforms.Therefore,it is necessary to delve deeper into sarcasm detection in multimodal Chinese content,specifically within images and text.To address this need,we propose a multimodal Chinese sarcasm detection model called the fuse cross-attention model(FCAM).This model incorporates a cross-attention mechanism to identify inconsistencies between modes.The text convolutional neural network(TextCNN)is used to extract basic features of Chinese text,while the deep residential network(ResNet)is used to extract image features.The cross-attention mechanism is used to obtain attention features from the text and image layers.The residual method is employed to establish a connection between the basic text features and the text layer’s attention features,as well as a link between the image features and the image layer’s attention features.These two feature representations are fused using the attention mechanism,resulting in the sarcasm classification results through the classification layer.We have constructed a multimodal Chinese sarcasm data set based on Weibo comment data related to the COVID-19 pandemic in a specific region.Experimental testing on this data set confirms that FCAM holds certain advantages over the benchmark model.