端到端增强特征神经网络的视频表情识别

Video Expression Recognition Based on End-to-End Enhanced Feature Neural Network

以卷积神经网络结合循环神经网络搭建端到端的深度学习网络,提出一种增强特征的视频表情识别方法。其中,卷积神经网络采用传统的VGG-16-FACE模型作为初始模型,完成特征提取;循环神经网络采用具有记忆能力的长期短时记忆模型网络(LSTM)结合连续视频的帧间信息给出最优预测。首先,对VGG-16和LSTM模型分别进行独立训练。因预测结果很大程度取决于LSTM模型,针对LSTM的层数和输出神经元个数进行优化调试后,得到两层LSTM,发现输出维度为2 048时识别效果最好。考虑到增加负责特征提取的VGG模型对预测结果的影响比重,模型由独立训练2个模型连接为端到端的1个模型。实验中考虑到1层LSTM输出会造成特征丢失,在端到端模型的基础上加入跳层连接,增强特征输入,最终实验结果表明:在AFEW数据集上对视频表情识别的准确率从32. 88%提升到37. 34%,F1分数从0. 289 5提升到0. 339 9,证实了端到端增强特征混合神经网络的有效性。

In this paper,a convolutional neural network( CNN) combined with a recurrent neural network( RNN) is built into an end-to-end deep learning network,and a video feature recognition method with enhanced features is proposed. The traditional VGG-16-FACE model is used as the initial model to complete the feature extraction. The long-term short-term memory( LSTM) model network with memory capability is combined with the inter-frame information of continuous video to make the optimal prediction. At the beginning of the experiment, the VGG and LSTM models were independently trained,and the prediction results largely depended on LSTM. We optimized the number of layers and output neurons for LSTM,and then theexperiments show that the results are better when the two-layer LSTM and its output are both 2 048. Then considering the increase of the influence of the VGG model responsible for feature extraction on the prediction results,the model is connected by an independent training model to an end-to-end model. In addition,the loss of features is caused by one layer of LSTM. The cross-layer connection is added on the basis of the end-to-end model to enhance feature input. The final experimental result improves the accuracy of video expression recognition from 32. 88% to 37. 34% and F1 score from 0. 289 5 to 0. 339 9 on the AFEW data set. It also confirmed the effectiveness of the end-to-end enhanced feature hybrid neural network.