一种文本-图像增强的多模态知识图谱嵌入方法

Multimodal Knowledge Graph Embedding with Text-Image Enhancement

大多传统的知识表示学习方法只关注三元组中的结构化信息,无法很好地利用实体图像、关系路径、文本描述等附加信息来学习知识表示或只融合一种附加信息。因此,提出同时融合实体描述和图像的多模态知识图谱嵌入方法,通过文本、图像相互增强,可以提供更加全面的外部信息来弥补单个信息源的不完整性给知识表示学习带来的不足。首先进行实体描述和图像建模,得到实体的文本表示和图像表示,并把它们作为TransE中结构表示的补充,最后通过3种实体表示的联合训练实现知识图谱和文本、图像的统一空间表示,提高实体和关系预测的准确性。实验结果表明,该模型实体预测的命中率比不融合附加信息的方法提高了3.09%,比只融合实体描述的方法提高了0.97%,比只融合实体图像的方法提高了1.32%。

Most traditional knowledge representation learning methods only focus on the structured information in triples,and cannot make good use of the additional information such as entity images,relation path and text descriptions to learn knowledge representation or fuse only one additional information.Therefore,a multimodal knowledge graph embedding method combining entity descriptions and images is proposed.Through mutual enhancement of text and images,more comprehensive external information can be provided to make up for the deficiency of knowledge representation learning caused by the incompleteness of a single information source.Firstly,text representation and image representation of entities are obtained by modeling entity descriptions and images.Then,they are used as a supplement to the structural representation in TransE.Finally,through the joint trai-ning of three entity representations,the unified spatial representation of knowledge graph,text and image is realized to improve the accuracy of entity and relation prediction.Experimental results show that the hit rate of entity prediction of this model improves by 3.09%compared with the method of without additional information,improves by 0.97%compared with the method of fusing only entity descriptions,and improves by 1.32%compared with the method of fusing only entity images.