融合注意力机制和迁移学习的跨数据集微表情识别

Cross-database micro-expression recognition combining attention mechanism and transfer learning

针对传统光流法泛化能力差,训练过程极易出现过拟合,造成微表情识别率不高等问题,在特征提取阶段,根据残差结构思想,在每层金字塔级采用独立且序列化的方式训练卷积网络,结合注意力机制对微表情图像的光流矢量表达进行逐级细化,构建一种基于关键帧的金字塔光流模型,与三正交平面的局部二值模式(LBP-TOP)特征级联融合得到最终特征表示,有效提取了视频序列的时空纹理特征及光学应变信息;在分类模型方面,针对深度学习识别模型应用于微表情分类时,由于卷积神经网络(CNN)无法实现面部关键区域与对应情感标签向量紧密关联导致分类性能较差的问题,提出一种以CNN为主体,结合图卷积网络(GCN)的微表情跨数据集迁移学习网络框架,对图像融合特征和标签向量隐藏联系进行分析,利用宏表情定量优势辅助微表情识别,在CASMEⅡ和SAMM两种微表情数据集上实现4种情绪数据的分类,识别率从57.56%升至75.93%。

Factors like a poor generalization ability and overfitting in the training process of the traditional optical flow method cause a low recognition rate of micro-expression. In the feature extraction stage, the convolution network is trained independently and serially at each pyramid level according to the idea of residual structure, and an optical flow vector expression of micro-expression images is refined step by step by combining attention mechanism so as to construct a pyramid optical flow model based on key frames. The final feature representation is obtained by cascading fusion with local binary patterns from three orthogonal planes(LBP-TOP) features, which effectively extracts the spatio-temporal texture features and optical strain information of video sequences. In the aspect of classification model, when the deep learning recognition model is applied to micro-expression classification, convolutional neural network(CNN) cannot achieve a close association between the key facial regions and the corresponding emotional tag vectors, which leads to poor classification performance. Therefore, this paper proposes a transfer learning network framework for micro-expression cross datasets based on CNN and graph convolutional network(GCN) to analyze the concealed relationship between image fusion features and tag vectors, which uses the quantitative advantages of macro-expression to assist micro-expression recognition. Experiments are carried out on two micro-expression databases, CASME II and SAMM, to realize the classification of four kinds of emotion data, and the recognition rate rises from 57.56% to 75.93%.