基于模态注意力图卷积特征融合的EEG和fNIRS情感识别

EEG and fNIRS emotion recognition based on modality attentiongraph convolution feature fusion

为了提升情感识别的准确率,从情绪视频引起的脑电(EEG)和功能近红外(fNIRS)数据中提取每个通道的信号之间的联系,并提出基于模态注意力多路图卷积神经网络(MA-MP-GF)的特征融合情感识别方法.将EEG和fNIRS数据构建为图结构数据,通过多路图卷积分别对每种模态的信号进行特征提取;利用模态注意力图卷积层融合不同模态通道间的连接信息.模态注意力机制可以赋予不同模态节点不同权重,使得图卷积层能够更加充分提取不同模态节点间连接关系.对采集的30个被试的4类情感数据进行实验测试,与仅EEG和仅fNIRS单模态识别结果相比,所提出的图卷积融合方法能够获得更高的识别准确率,分别提升了8.06%、22.90%;与当前常用的EEG-fNIRS融合方法相比,所提出的图卷积融合方法的平均识别准确率提升了2.76%~7.36%;图卷积融合方法在加入模态注意力后识别率最高提升了1.68%.

A feature fusion emotion recognition method based on modality attention multi-path convolutional neural network was proposed,extracting the connection between the signals of each channel from the electroencephalogram(EEG)and functional near infrared spectroscopy(fNIRS)data induced by emotional video to improve the accuracy of emotion recognition.The EEG and fNIRS data were constructed as graph structure data,and the feature of each mode signal was extracted by multi-path graph convolution.The information of connection between different modal channels was fused by modality attention graph convolution.The modality attention mechanism can give different weights to different modal nodes,thus the graph convolution layer can more fully extract the connection relationship between different modal nodes.Experimental tests were carried out on four types of emotional data collected from 30 subjects.Compared with the results of EEG only and fNIRS only,the recognition accuracy of the proposed graph convolution fusion method was higher,which increased by 8.06%and 22.90%respectively.Compared with the current commonly used EEG and fNIRS fusion method,the average recognition accuracy of the proposed graph convolution fusion method was improved by 2.76%~7.36%.The recognition rate of graph convolution fusion method increased by 1.68%after adding modality attention.