基于语音、表情与姿态的三模态普通话情感识别

Tri-modal Mandarin Emotion Recognition Based on Speech,Facial Expression and Body Gesture

为了克服单模态普通话情感识别效率低、可靠性差的问题,对情感识别非常重要的语音、表情与姿态三个模态进行特征层融合,提出普通话三模态情感识别方法。首先从语音声学、面部表情和运动姿态中分别提取不同的情感特征,然后采用改进的遗传算法IAGA进行三模态情感特征融合,最后利用SVM分类器构建预测模型并完成情感识别。将三模态融合方法在CHEAVD中文多模态情感数据集进行测试,并和传统的单模态、双模态情感识别进行对比,实验结果表明,双模态融合的情感识别率要高于单模态,三模态融合的情感识别率高于双模态,验证了普通话三模态情感识别方法的有效性。

In order to overcome the problem of low efficiency and poor reliability of single-modal mandarin emotion recognition,three modes such as speech,facial expression and body gesture are combined,which are very important to emotion recognition,and a tri-modal mandarin emotion recognition method is proposed.Firstly,speech,facial expression and body gesture are extracted from emotional data samples,and then the three modal emotion features are fused by improved adaptive genetic algorithm(IAGA)method.Finally,SVM classifier is used to construct prediction model and emotion recognition is completed on it.The tri-modal fusion method is tested in Chinese natural emotional audio-visual database(CHEAVD),and is compared with traditional single-modal and bi-modal emotion recognition,the experimental results show that the dual-modal fusion has higher emotion recognition rate than the mono-modal emotion recognition,and the tri-modal fusion has higher emotion recognition rate than the dual-modal emotion recognition,which verifies the effectiveness of the tri-modal mandarin emotion recognition methods.