灾害事件下基于多模态融合的社交媒体信息丰富型推文识别研究

Research on the Identification of Disasters Related Informative Tweets Based on Multimodal Fusion in Social Media

【目的/意义】灾害事件期间,反映事件实时状态的多模态信息被广泛发布在社交媒体平台中,有效识别这些蕴含丰富灾害信息的推文对于灾害事件的危机预警、态势感知和应急决策都具有重要的作用。【方法/过程】本文提出一种基于多模态融合的信息丰富型推文识别框架,将推文中图文的形式特征和语义特征作为融合输入,利用机器学习和深度学习的方法构建多种识别信息丰富型推文的二元分类器,进而分析不同推文特征和不同融合方法在信息丰富型推文识别中的效果。【结果/结论】实验结果表明,基于语义特征的识别方法要优于基于形式特征的识别方法,基于图文融合的识别方法优于基于单一模态文本或图像的识别方法,同时融合文本和图像的形式特征和语义特征来识别灾害事件信息丰富型推文的效果更好。【创新/局限】本文在综合考虑推文中多模态信息的形式特征和语义特征的基础上,提出基于特征层和决策层的多模态融合方法来识别社交媒体中的信息丰富型推文,但是该方法在不同灾害事件场景下的适用性还有待进一步检验。

【Purpose/significance】During the disasters,multimodal information reflecting the real-time situation is widely published in the social media.How to effectively identify informative tweets is important to disasters response.【Method/process】This paper proposes a social media informative tweets identification framework based on multimodal fusion.Formal features and semantic features of text and text in tweets are used as fusion inputs.Then,a binary classifier to identify informative is constructed through machine learning and deep learning.Finally,the role of different tweet features and fusion methods in informative tweets identification are analyzed.【Result/conclusion】The results of informative tweets identification experiment show that the effect of semantic features is better than formal features,and the effect of multimodal fusion are better than single modal such as text or images.At the same time,the best way of identifying informative tweets related disasters is fusing formal and semantic features of text and images.【Innovation/limitation】Considering the formal and semantic features of multimodal information in tweets,we propose a multimodal fusion method of feature layer and decision layer to identify informative tweets in social media.However,the applicability of this method in different disasters needs to be further tested.