多模态数据的洪涝灾害知识图谱构建与应用

Construction and Application of Flood Disaster Knowledge Graph Based on Multi-modal Data

洪涝灾害发生过程中观测数据多源异构(遥感影像、社交媒体文本、地理信息数据等),难以利用互补优势融合应用于风险评估和提供决策知识。研究基于多模态数据的洪涝灾害知识图谱构建方法,融合抽取遥感影像与社交媒体文本知识,形成多模态洪涝灾害知识图谱。基于自顶向下的方法细分领域概念,构建洪涝灾害领域本体层。通过深度残差全卷积神经网络对遥感影像进行智能解译,利用地理逆编码将影像解译信息转化为文本,实现影像信息到文本知识的转化。基于命名实体识别技术与关系抽取技术对社交媒体文本数据进行知识抽取。通过训练词向量,利用语义相似度计算关联文本知识与影像知识,实现多模态数据知识统一表达。以中国湖北省洪涝灾害为例,该方法将多源异构的数据高效转化为知识并进行关联,形成领域知识图谱,实现了多源异构数据到多模态知识的转化。在灾害不同时期提供相应应急措施,并且通过关联农业受灾面积、农作物类型、农作物价值实现湖北省洪涝灾害评估。该方法结合深度遥感解译、文本知识抽取技术以及语义相似度计算,实现了多源异构数据到多模态知识的转化。

Objectives:In the process of flood disaster,the observed data is multi-source heterogeneous(remote sensing image,social media text,geographic information data,etc.),which makes it difficult to use complementary advantages to fuse and apply to risk assessment and provide decision-making knowledge.We study the construction method of flood disaster knowledge graph based on multi-modal data.Remote sensing images and social media text knowledge are integrated and extracted to form multi-modal flood disaster knowledge graph.Methods:Based on the top-down method,the domain concept is subdivided and the flood disaster domain ontology layer is constructed.Remote sensing images are intelligently interpreted by deep residual full convolutional neural network,and the image interpretation information is converted into text by geographic inverse coding to realize transformation from image information to text knowledge.Knowledge is extracted from social media textual data based on naming entity recognition and relationship extraction technique.By training word vectors,the semantic similarity is used to calculate the correlation between text knowledge and image knowledge,so as to achieve the unified expression of multi-modal data knowledge.Results:Taking the flood disaster in Hubei Province,China as an example,the multi-source heterogeneous data are efficiently transformed into knowledge and correlated to form domain knowledge graph,which realizes the transformation from multi-source heterogeneous data to multi-modal knowledge,provides corresponding emergency measures in different periods of disasters,and realizes flood disaster assessment in Hubei Province by associating agricultural disaster area,crop type and crop value.Conclusions:This method combines deep remote sensing interpretation,text knowledge extraction and semantic similarity calculation to realize transformation from multi-source heterogeneous data to multi-modal knowledge.