基于多模态信息融合的用户连接方法

Multi-modal information fusion for user identity linkage

为弥补用户连接现有工作中存在的以下两个问题:大部分方法只考虑用户的文本和网络结构信息,忽略了图像等多模态信息;现有方法大都通过简单的特征拼接融合不同模态的信息,忽略了这些信息的互补性、关联性和异质性,提出一个基于多模态信息融合的用户连接模型MIFUIL。获取用户的文本、视觉以及异构网络的嵌入表示;利用注意力机制学习不同模态信息间的互补性、关联性和异质性,获取用户多模态信息的融合嵌入表示,通过多模态对比学习实现跨平台的用户连接。实验结果表明,MIFUIL模型在两个多模态数据集TWFQ、DB-YAGO上的性能均优于现有方法。

To address the following two problems of existing work for user identity linkage(UIL),one is that only text and network structure modal information is considered in most methods,more modal information such as images is ignored,the other is that multi-modal information is fused through simple characteristic concatenation,while the complementarity,correlation,and heterogeneity of different modal information is ignored,an UIL method based on multi-modal information fusion,named MIFUIL,was proposed.The embedding of text,visual and heterogeneous network modal information of the user was acquired.The attention mechanisms was used to learn the complementarity,correlation,and heterogeneity among different modal information to obtain the fused embedding of the user,thus identical user identities across platforms were linked through multi-modal contrastive learning.Extensive experiments on two real-world multi-modal datasets TWFQ and DB-YAGO demonstrate that MIFUIL significantly outperforms the state-of-art methods.