基于EEG和面部视频的多模态连续情感识别

Multimodal continuous emotion recognition based on EEG and facial video

针对脑电(Electroencephalogram, EEG)通道间和时间上情绪强度的改变很难被捕捉,以及不同被试的面部特征情绪上的相似性难以挖掘的问题,文章提出了一种基于EEG和面部视频的多模态连续情感识别模型.采用基于时空注意力机制(Spatial-Temporal Attention)的卷积和双向长短期记忆神经网络的组合模型(STA-CNNBiLSTM)对EEG中提取的功率谱密度(Power Spectral Density, PSD)特征进行深层特征学习与情感分类;采用引入自注意力机制的预训练卷积神经网络(SA-CNN)对人脸面部几何特征进行学习与情感分类.采用决策级融合算法,对两个模态的分类结果进行迭代学习与融合,得到最终多模态情感分类结果.在公开数据集MAHNOB-HCI进行了大量对比验证实验,在FER2013数据集的面部几何特征上对SA-CNN模型进行了预训练.在独立被试的实验中,所提模型在效价维度二分类的平均准确率为75.50%,在唤醒维度二分类的平均准确率为79.00%,均优于单模态上的最高平均准确率.和目前流行的模型LSSVM、SE-CNN和AM-LSTM相比较,所提模型的分类效果更优,验证了所提时空注意力机制能够捕捉更多的EEG时空特征,自注意力机制能够关注到不同被试面部特征的相似性,进而提高了多模态情感识别的性能.

To solve the difficulty in capturing the changes of emotional intensity between EEG channels and time points,and the difficulty in capturing the emotional similarity of facial features of different subjects,this paper proposes a multimodal continuous emotion recognition model based on EEG and facial videos.A combined model of convolution and bidirectional short-term memory neural network based on spatiotemporal attention(STA-CNNBiLSTM)is used to learn and classify deep emotion related dynamics from the power spectral density(PSD)features of EEG.A pre-training convolutional neural network with self-attention(SA-CNN)is used to learn and classify the deep facial geometric features.We use the decision level fusion algorithm to fuse the above two classification results of EEG and facial geometric features modes to obtain the final results.A large number of comparative verification experiments were carried out on the public MAHNOB-HCI dataset,and the SA-CNN model was pre-trained on the facial geometric features of the FER2013 data set.In the subject-independent experiment,the average binary classification accuracy of the proposed model achieved 75.00%in valence and 79.00%in arousal,both of which are better than that in each single mode.Compared with the current popular models SE-CNN and AM-LSTM,the classification performance of the proposed model is also better,which proved that the proposed spatiotemporal attention mechanism can capture more spatiotemporal EEG features,and the self-attention mechanism can focus on the similarity of facial features of different subjects,thus improving the performance of multimodal emotion recognition.