基于长短记忆与信息注意的视频–脑电交互协同情感识别

Video-EEG Based Collaborative Emotion Recognition Using LSTM and Information-Attention

基于视频–脑电信号交互协同的情感识别是人机交互重要而具有挑战性的研究问题.本文提出了基于长短记忆神经网络(Long-short term memory,LSTM)和注意机制(Attention mechanism)的视频–脑电信号交互协同的情感识别模型.模型的输入是实验参与人员观看情感诱导视频时采集到的人脸视频与脑电信号,输出是实验参与人员的情感识别结果.该模型在每一个时间点上同时提取基于卷积神经网络(Convolution neural network,CNN)的人脸视频特征与对应的脑电信号特征,通过LSTM进行融合并预测下一个时间点上的关键情感信号帧,直至最后一个时间点上计算出情感识别结果.在这一过程中,该模型通过空域频带注意机制计算脑电信号α波,β波与θ波的重要度,从而更加有效地利用脑电信号的空域关键信息;通过时域注意机制,预测下一时间点上的关键信号帧,从而更加有效地利用情感数据的时域关键信息.本文在MAHNOB-HCI和DEAP两个典型数据集上测试了所提出的方法和模型,取得了良好的识别效果.实验结果表明本文的工作为视频–脑电信号交互协同的情感识别问题提供了一种有效的解决方法.

Video-EEG based collaborative emotion recognition is an important yet challenging problem in research of human-computer interaction. In this paper, we propose a novel model for video-EEG based collaborative emotion recognition by virtue of long-short term memory neural network(LSTM) and attention mechanism. The inputs of this model are the facial videos and EEG signals collected from a participant who is watching video clips for emotional inducement. The output is the participant s emotion states. At each time step, the model employs convolution neural network(CNN) to extract features from video frames and corresponding EEG slices. Then it employs LSTM to iteratively fuse the multi-modal features and predict the next key-emotion frame until it yields the emotion state at the last time step. Within the process, the model computes the importance of different frequency-band EEG signals, i.e. α wave, βwave, and θ wave, through spatial band attention, in order to effectively use the key information of EEG signals. With the temporal attention, it predicts the next key emotion frame in order to take advantage of the temporal key information of emotional data. Experiments on MAHNOB-HCI dataset and DEAP dataset show encouraging results and demonstrate the strength of our model. The results show that the proposed method presents a different perspective for effective collaborative emotion recognition.