基于时序感知DAG的多模态对话情绪识别模型

Multi-modal temporal-aware DAG for emotion recognition in conversation

针对现有对话情绪识别方法中对时序信息、话语者信息、多模态信息利用不充分的问题,提出了一个时序信息感知的多模态有向无环图模型(MTDAG)。其中所设计的时序感知单元能按照时间顺序优化话语权重设置,并收集历史情绪线索,实现基于近因效应下对时序信息和历史信息更有效的利用;设计的上下文和话语者信息融合模块,通过提取上下文语境和话语者自语境的深度联合信息实现对话语者信息的充分利用;通过设置DAG(directed acyclic graph)子图捕获多模态信息并约束交互方向的方式,在减少噪声引入的基础上充分利用多模态信息。在两个基准数据集IEMOCAP和MELD的大量实验表明该模型具有较好的情绪识别效果。

Aiming at the issue of insufficient utilization of temporal information,speaker information,and multi-modal information in existing conversational emotion recognition methods,this paper proposed a multi-modal temporal-aware DAG model(MTDAG).The designed temporal-aware unit optimized the discourse weight setting in chronological order and collected historical emotional cues to achieve more effective utilization of temporal and historical information based on recency effect.The context and speaker information fusion module achieved the full utilization of discourse information by extracting the deep joint information of contextual context and speaker self-context.By setting the DAG subgraphs to capture multi-modal information and constrain the interaction direction,the model achieved full utilization of multi-modal information while reducing the introduction of noise.Extensive experiments conducted on two benchmark datasets,IEMOCAP and MELD,demonstrate that the model exhibits excellent performance in emotion recognition.