基于度量学习的多模态谣言检测

Metric learning-based multimodal rumor detection

目前主流的多模态谣言检测模型,主要侧重于建模过程中模态的特征提取与拼接方法研究,而各模态局部特征关系、模态内与模态间的信息交互往往被忽略,这在一定程度上影响到了谣言检测的效果。针对该问题,提出了一种基于度量学习的多模态谣言检测方法。考虑到各模态局部特征关系对模态整体特征表示的影响,采用了句法分析和注意力机制技术分别挖掘文本和图片的局部特征关系;同时,将度量学习应用到谣言检测中,通过三元组学习和对比学习找出模态内与模态间的关联信息。在Twitter和Weibo两个公开的数据集上进行了性能测试实验,准确率分别达到92.8%和85.2%,结果表明将各模态局部特征关系、模态内与模态间的信息交互加入谣言检测模型中能够进一步提升谣言检测的精准度。

At present,the mainstream multi-modal rumor detection models mainly focus on the feature extraction and splicing methods of the modes in the modeling process,while the local feature relationship of each mode and the information interaction within and between modes are often ignored,which affects the effect of rumor detection to a certain extent.To address this issue,we propose a metric learning-based multimodal rumor detection method.Considering the influence of local feature relationships within each modality on the overall representation of modalities,we employed the technology of syntactic analysis and attention mechanism to exploring the local feature relationships of text and images,respectively.Additionally,metric learning is applied to rumor detection,where triplet learning and contrastive learning are utilized to identify the associated information within and between modalities.Performance testing experiments conducted on publicly available datasets from Twitter and Weibo demonstrate accuracy rates of 92.8%and 85.2%,respectively.These results indicate that incorporating local feature relationships within each modality and the interaction between modalities into the rumor detection model can further enhance the accuracy of rumor detection.