基于多模态表示学习的情感分析框架

Sentiment Analysis Framework Based on Multimodal Representation Learning

在多模态表示对整体损失的学习过程中,重构损失对模型的依赖性相对较小,导致隐含表示无法有效捕捉它们各自模态的细节。文中提出了一个基于多模态表示学习的多子空间情感分析框架。首先将每个模态投射到模态不变和模态特定两种不同的话语表示中,在模态不变表示中构建主共享子空间以及帮助该子空间减少模态差距的辅助共享子空间,在模态特定表示中构建私有子空间以捕获每个模态独有的特征,将所有子空间中的隐藏向量作为解码函数的输入并重构模态向量,以实现对重构损失的优化。然后,在融合阶段对每个模态表示执行基于Transformer的自注意力,使每个表示能从对整体情感取向具有协同作用的其他跨模态表示中获取潜在信息。最后,通过串联生成联合向量并利用全连接层生成任务预测。在两个公开数据集MOSI和MOSEI上的实验结果表明,该框架在大多数评价指标上都优于基线模型。

In the process of learning the overall loss of multimodal representations,the dependence of reconstruction loss on the model is relatively less,resulting in hidden representations that cannot effectively capture the details of their respective modalities.This paper proposes a multi-subspace sentiment analysis framework.Firstly,the framework projects each modality to two distinct utterance representations:modality-invariant and modality-specific.We construct the main shared subspace and the auxiliary shared subspace that helps the main subspace to reduce the modality gap in the modality-invariant representation.Also,construct the private subspaces in the modality-specific representation to capture the characteristic features of each modality.We take the hidden vectors in all subspaces as the input of the decoder function and reconstruct the modal vector to achieve optimization of reconstruction loss.Secondly,in the fusion procedure,we perform a multi-headed self-attention based on Transformer on these representations,so that each cross-modal representation can induce potential information from fellow representations that have a synergistic effect on the overall emotional orientation.Finally,we construct a joint-vector by using concatenation and use fully connected layers to generate task predictions.Experimental results on both MOSI and MOSEI datasets show that the proposed framework outperforms the baselines in most evaluation criteria.