基于姿态校正与姿态融合的2D/3D骨架动作识别方法

2D/3D skeleton action recognition based on posture transformation and posture fusion

针对现有的人体骨架动作识别方法对肢体信息挖掘不足以及时间特征提取不足的问题,提出了一种基于姿态校正模块与姿态融合模块的模型PTF-SGN,实现了对骨架图关键时空信息的充分利用。首先,对骨架图数据进行预处理,挖掘肢体和关节点的位移信息并提取特征;然后,姿态校正模块通过无监督学习的方式获取姿态调整因子,并对人体姿态进行自适应调整,增强了模型在不同环境下的鲁棒性;其次,提出一种基于时间注意力机制的姿态融合模块,学习骨架图中的短时刻特征与长时刻特征并融合长短时刻特征,加强了对时间特征的表征能力;最后,将骨架图的全局时空特征输入到分类网络中得到动作识别结果。在NTU60 RGB+D、NTU120 RGB+D两个3D骨架数据集和Penn-Action、HARPET两个2D骨架数据集上的实验结果表明,该模型能够有效地识别骨架时序数据的动作。

Aiming at the problems that existing human skeleton action recognition methods couldn’t explore sufficient human body information and extract sufficient temporal feature, this paper proposed a model based on posture transformation module and posture fusion module(PTF-SGN),which realized the utilization of the key spatio-temporal information in skeleton diagram.Firstly, by preprocessing the skeleton diagram, the model mined the displacement information of limbs and joints, and extracted the features.Then it used the posture transformation module to obtain the posture adjustment factors from the skeleton image data in an unsupervised learning manner, and adaptively adjusted the body posture to enhance the robustness of the model in different environments.Secondly, it proposed a posture fusion module based on the time attention mechanism, which learned the short-term features and the long-term features, and fused the time characteristics of long and short moments to strengthen the characterization ability of time characteristics.Finally, it extracted the global spatio-temporal feature of the skeleton feature to input into the classification network to obtain the action recognition result.The experimental results on the two 3D skeleton datasets of NTU60 RGB+D and NTU120 RGB+D and the two 2D skeleton datasets of Penn-Action and HARPET show that PTF-SGN model can effectively recognize actions of skeleton time series data.