基于2D转3D骨架的多特征融合实时动作识别

Multi-Feature Fusion Real-Time Action Recognition Based on 2D to 3D Skeleton

提出了一种基于二维(2D)转三维(3D)骨架的实时检测双分支子网络,可实现2D骨架关键点的3D估计和2D、3D骨架特征融合的人体3D动作识别。在检测过程采用OpenPose框架实时获取视频中人体骨架的2D关键点坐标。在2D转3D骨架估计过程中,设计了一种输入为难样本且具有反馈功能的孪生网络。在3D动作识别过程中设计了一种2D、3D骨架特征双分支孪生网络,以完成3D姿态识别任务。在Human3.6M数据集上训练3D骨架估计网络,在基于欧拉变换的NTU RGB+D 60多视角增强数据集上训练骨架动作识别网络,最终得到的3D骨架动作识别交叉受试者准确率为88.2%,交叉视野准确率为95.6%。实验结果表明,该方法对3D骨架的预测精度较高,且具有实时反馈能力,可适用于实时监控中的动作识别。

We propose a real-time detection binary sub network based on two-dimensional(2 D) to three-dimensional(3 D) skeleton, which can realize 3 D estimation of key points of 2 D skeleton and human 3 D motion recognition based on 2 D and 3 D skeleton feature fusion. In the detection process, OpenPose framework is used to obtain the 2 D key point coordinates of human skeleton in video in real time. In the process of 2 D to 3 D skeleton estimation, a siamese network with difficult input samples and feedback function is designed. In the process of 3 D motion recognition, a two branch siamese network of 2 D and 3 D skeleton features is designed to complete the task of 3 D pose recognition. The 3 D skeleton estimation network is trained on the Human3.6 M data set, and the skeleton action recognition network is trained on the NTU RGB+D 60 multi view enhancement data set based on Euler transform. Finally, the accuracy of cross subjects and accuracy of cross views are 88.2% and 95.6%. Experimental results show that the method has high prediction accuracy for 3 D skeleton and real-time feedback ability, and can be applied to action recognition in real-time monitoring.