基于Android的多模态情感识别APP的设计与开发

Design and development of multi⁃modal emotion recognition APP based on Android

在团队自建的中文多模态情感识别语料库的基础上,训练了多模态情感识别深度神经网络模型,能综合视频中图片、音频和文本三个模态信息进行情感识别。基于该情感识别模型,设计并开发了一款基于Android平台的移动应用程序,捕获视频中人物的话语文本、声音和面部表情,识别出人物的情感状态,以支撑后续共情聊天机器人的研发。

On the basis of the Chinese multi⁃modal emotion recognition corpus built by the team,a deep neural network model for multi⁃modal emotion recognition is trained,which can synthesize the picture,audio and text modal information in video for emo⁃tion recognition.Based on the emotion recognition model,a mobile application based on the Android platform is designed and de⁃veloped,which captures the speech text,voice and facial expression of the characters in the video,recognizes the emotional state of the characters,and supports the research and development of the follow⁃up empathic chat robot.