面向多模态知识图谱的实体对齐方法研究

Entity Alignment on Multi-modal Knowledge Graph

实体对齐是构建知识图谱的重要环节,也是该领域的一个研究热点.现有实体对齐工作在包含文本、图片的多模态知识图谱数据集DB15K-FB15K和YAGO15K-FB15K上做了大量研究,但是它们仅局限于文本和图片两种模态,且在多模态知识融合方面的性能并不显著.为弥补已有工作的不足,本文构建了一个包含文本、图片、视频的多模态知识图谱数据集Douban-Baidu,并提出了EA-MMKG模型来解决多模态知识图谱实体对齐问题.EA-MMKG包含两部分:多模态知识嵌入模块和多模态知识交互融合模块.具体来讲,多模态知识嵌入模块由关系三元组嵌入、图片嵌入、视频嵌入和属性三元组嵌入4个部分组成;多模态知识交互融合模块采用了基于注意力的融合机制来融合从文本、图片、视频3种模态中提取的特征信息,从而使得各模态之间的交互更加充分、融合效果更好,并最终提高多模态知识图谱实体对齐的性能.实验结果表明,EA-MMKG模型在Douban-Baidu数据集、DB15K-FB15K数据集和YAGO15K-FB15K数据集上的性能均优于现有的模型.

Entity alignment plays an essential role in the construction of knowledge graphs,and has attracted increasing attention recently.Existing work has made great efforts in entity alignment based on the publicly available datasets DB15K-FB15K and YAGO15K-FB15K.However,these two datasets only contain the text and images,and the current work cannot achieve high performance on multi-modal knowledge fusion.To overcome the shortcomings of existing studies,we construct a multi-modal knowledge graph dataset named Douban-Baidu,which includes the text,images and videos.A novel model called EA-MMKG is proposed to solve the problem of multi-modal knowledge graph entity alignment.The EA-MMKG model consists of a multi-modal knowledge embedding module and a multi-modal knowledge interaction fusion module.Specifically,the multi-modal knowledge embedding module includes the embeddings of relationship triples,images,videos and attribute triples.The multi-modal knowledge interaction fusion module adopts an attention mechanism to combine the feature information extracted from the text,images and videos.This module makes the interaction between modalities more sufficient and further improves the performance of multi-modal knowledge graph entity alignment.In this paper,the experiments on datasets Douban-Baidu,DB15K-FB15K and YAGO15K-FB15K demonstrate that the proposed model EA-MMKG outperforms the state-of-the-art methods.