基于注意力机制和特征融合的手势识别方法

GESTURE RECOGNITION BASED ON ATTENTION MECHANISM AND FEATURE FUSION

动态手势视频流预处理过程中,随机采样或密集采样存在关键帧丢失或数据冗余的问题,导致特征融合在单个特征的时序建模中,可能丢失重要的时序信息,由此提出基于注意力机制和特征融合的手势识别方法。通过含注意力机制的长短期记忆网络,在时序建模过程中抽取重要数据,有效避免了采样方法的随意性或盲目性;设计具有三层结构的特征融合网络对抽取的RGB特征和深度图像特征进行融合处理,提升了动态手势识别的准确率。实验结果表明引入注意力机制的必要性,验证了特征融合的有效性和该方法的鲁棒性。

In the preprocessing of dynamic gesture video stream,random or dense sampling may lead to key frames missing or data redundancy,resulting in feature fusion in the temporal modeling of single feature,and losing important temporal information.Therefore,we propose a gesture recognition method based on attention mechanism and feature fusion.The important data were extracted in the process of temporal modeling using the LSTM with attention mechanism which effectively avoided the randomness or blindness of the sampling method;a feature fusion network with three-layer structure was designed to fuse the extracted RGB features and depth image features,which improves the accuracy of dynamic gesture recognition.The experimental results show the necessity of introducing attention mechanism,and verify the effectiveness of feature fusion and the robustness of our method.