基于多源域对抗迁移学习的可穿戴情绪识别技术

Research on Wearable Emotion Recognition Based on Multi-Source Domain Adversarial Transfer Learning

情绪影响身心健康及认知功能等,因而在人们的生活中扮演着重要角色.自动情绪识别有助于预警心理疾病和探索行为机制,具有巨大的研究与应用价值.在过去十余年中,研究者们提出了各种情绪识别方法,但均存在不同方面的不足:基于脑电图(Electroencephalography,EEG)信号的方法需采用专业、昂贵且不易操作的脑电仪;基于视觉和语音的方法存在隐私泄露的风险;基于手机使用模式分析的方法其可靠性和准确性有待提高等.本文利用生理信号如呼吸音、心跳音及脉搏等与情绪的潜在关联性,创新性地提出基于低成本、普适易用可穿戴硬件的情绪识别技术,借助多模态数据融合对不同类型数据进行有效利用,既减少了数据冗余又有效提升了系统性能.此外,在保证良好识别准确率的前提下,为提升情绪识别模型对不同用户的泛化性、最大化降低新用户的使用成本,本文提出了基于多源域对抗思想的情绪识别模型,借助少量来自新用户的无标签数据实现模型的无监督迁移,再辅之以极少量有标签数据微调分类器参数可进一步提升情绪识别准确率.为验证所提情绪识别方法的有效性,本文设计并实现了一套融合麦克风与光电容积脉搏波(Photoplethysmography,PPG)传感器以测量人体心跳音、呼吸音及脉搏等生理指征的可穿戴系统.基于此系统,本文在不同设置下开展了大量实验并对不同影响因素进行了评估.实验结果表明:对于四类基本情绪,本文所提方法单被试识别准确率可达95.0%,跨被试识别准确率为62.5%,比基准方法提升了5.3%.结合有监督小样本参数微调,识别准确率可进一步提高至81.1%,比基准方法提高了12.4%.上述结果验证了本文所提方法的可行性,为泛在情绪识别研究做出了崭新的探索.

Emotions can profoundly impact both human’s overall well-being and cognitive function.As a result,they are of paramount significance in the realm of human life especially in modern society with increasing pressures.Automatic emotion recognition contributes to early warning of psychological disorders and the exploration of behavioral mechanisms,holding immense research and practical value.Over the past decade,researchers have proposed various kinds of methods for automatic emotion recognition based on different sensing mechanisms.Nevertheless,each of them exhibits deficiencies in different aspects.For example,the methods based on electroencephalogram(EEG)signals require the use of specialized,costly,and challenging-to-operate EEG devices;the methods relying on visual and speech cues carry privacy risks;and the methods based on the analysis of mobile phone usage pattern need improvement in terms of reliability and accuracy.Considering the above,this paper proposes a novel approach to automatic emotion recognition that utilizes low-cost,readily available,and easy-to-use wearable hardware.In a detail,this paper makes use of the potential correlations between physiological signals,namely,breathing and heartbeat sounds,and pulse with human emotions.By employing data fusion across multiple sensing modalities,this work effectively harnesses diverse information types,reducing data redundancy,and substantially improving the system performance at the same time.Furthermore,while ensuring a high recognition accuracy,this paper also proposes an emotion recognition model based on a multi-source domain adversarial approach which aims to enhance the generalization of emotion recognition across diverse users and minimize the cost for unseen users.Our method first leverages a small amount of unlabeled data from unseen users to achieve quick model adaptation in an unsupervised approach,and then fine-tune the classifier’s parameters with a minimal amount of labeled data to further improve emotion recognition accuracy.To validate the effectiveness of our proposed emotion recognition method,this paper designs and implements a wearable system that integrate two microphones and photoplethysmography(PPG)sensors to measure physiological signs.Among them,the two microphones are equipped in a smartglasses and earphone to collect sounds produced by heartbeats and breathing,respectively;the two PPG sensors are embedded in the smartglasses and a smartwatch to measure the blood pulses in the head and wrist,respectively.Based on this wearable system,we have conducted extensive experiments in diverse settings with thirty participants aged from 17 to 30 years old.We have also carried an assessment of the impact of different environmental factors such as noise,hardware,and wearing positions to evaluate the robustness of our emotion recognition system.The experimental results demonstrate that for the four basic emotions,the proposed method achieves an average recognition accuracy of 95.0%in the subject-dependent cases,and an average accuracy of 62.5%in the cross-subject cases after using multi-source domain adversarial transfer learning,with a 5.3%improvement over the baseline methods.When combined with supervised fine-tuning with few shots,the recognition accuracy further increases to 81.1%,surpassing the baseline methods by 12.4%.These findings affirm the feasibility of the proposed method and offer a fresh perspective for ubiquitous emotion recognition research.