基于3D分层卷积融合的多模态生理信号情绪识别

Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion

近年来,脑电等生理信号由于能客观体现真实情绪已逐渐成为情绪识别研究的热门对象。然而,单模态的脑电信号存在情绪信息特征不完备问题,多模态生理信号存在情绪信息交互不充分问题。针对这些问题,提出基于3D分层卷积的多模态特征融合模型,旨在充分挖掘多模态交互关系,更准确地刻画情感信息。首先通过深度可分离卷积网络提取脑电、眼电和肌电3种模态的生理信号的多模态初级情绪特征信息,再对得到的多模态初级情绪特征信息进行3D卷积融合操作,实现两两模态间的局部交互以及所有模态间的全局交互,获取包含不同生理信号情绪特征的多模态融合特征。实验结果表明,提出的模型在DEAP数据集的效价、唤醒度的二分类和四分类任务中达到了98%的平均准确率。

In recent years,physiological signals such as electroencephalograhpy(EEG)have gradually become popular objects of emotion recognition research because they can objectively reflect true emotions.However,the single-modal EEG signal has the problem of incomplete emotional information representation,and the multi-modal physiological signal has the problem of insufficient emotional information interaction.Therefore,a 3D hierarchical convolutional fusion model was proposed,which aimed to fully explore multi-modal interaction relationships and more accurately describe emotional information.The method first extracted the primary emotional representation information of EEG,electro-oculogram(EOG)and electromyography(EMG)by depthwise separable convolution network,and then performed 3D convolution fusion operation on the obtained multi-modal primary emotional representation information to realize the pairwise mode local interactions between states and global interactions among all modalities,so as to obtain multi-modal fusion representations containing emotional characteristics of different physiological signals.The results show that the accuracy in the valence and arousal of the two-class and four-class tasks on DEAP dataset are both 98%by the proposed model.