多模态与文本预训练模型的文本嵌入差异研究

Difference between Multi-modal vs.Text Pre-trained Models in Embedding Text

为了详细地分析文本单模态预训练模型RoBERTa和图文多模态预训练模型WenLan文本嵌入的差异,提出两种定量比较方法,即在任一空间中,使用距离一个词最近的k近邻词集合表示其语义,进而通过集合间的Jaccard相似度来分析两个空间中词的语义变化;将每个词与其k近邻词组成词对,分析词对之间的关系。实验结果表明,图文多模态预训练为更抽象的词(如成功和爱情等)带来更多的语义变化,可以更好地区分反义词,发现更多的上下义词,而文本单模态预训练模型更擅长发现同义词。另外,图文多模态预训练模型能够建立更广泛的词之间的相关关系。

This paper provides quantitative comparison between the text embedding of a text pre-trained model(i.e.,RoBERTa)and a multi-modal pre-trained model(i.e.,WenLan).Two quantitative comparison methods are proposed,in an embedding space:representing the semantics of a word using the set of􀝇-nearest words to it and then analyze the semantic changes of the word in the two spaces using the Jaccard similarity of the two sets;forming pairs between each word and its nearest􀝇words to analyze the relationship.The results show that the multi-modal pre-training brings more semantic changes for more abstract words(e.g.,success,love),and the multimodal pre-trained model can better differentiate antonyms and discover more hypernyms or hyponyms,while text pre-training works better in finding synonyms.Moreover,multi-modal pre-trained model can construct a more extensive associative relationship between words.