基于时间动态帧选择与时空图卷积的可解释骨架行为识别

Temporal dynamic frame selection and spatio-temporal graph convolution for interpretable skeleton-based action recognition

骨架行为识别是计算机视觉和机器学习领域的研究热点。现有数据驱动型神经网络往往忽略骨架序列时间动态帧选择和模型内在人类可理解的决策逻辑,造成可解释性不足。为此提出一种基于时间动态帧选择与时空图卷积的可解释骨架行为识别方法,以提高模型的可解释性和识别性能。首先利用骨架帧置信度评价函数删除低质骨架帧,以解决骨架序列噪声问题。其次基于人体运动领域知识,提出自适应时间动态帧选择模块用于计算运动行为显著区域,以捕捉关键人体运动骨架帧的动态规律。为学习行为骨架节点内在拓扑结构,改进时空图卷积网络用于可解释骨架行为识别。在NTU RGB+D,NTU RGB+D 120和FineGym这3个大型公开数据集上的实验评估表明,该方法的骨架行为识别准确率优于对比方法并具有可解释性。

Skeleton-based action recognition is a prominent research topic in computer vision and machine learning.Existing data-driven neural networks often overlook the temporal dynamic frame selection of skeleton sequences and lack the understandable decision logic inherent in the model,resulting in insufficient interpretability.To this end,we proposed an interpretable skeleton-based action recognition method based on temporal dynamic frame selection and spatio-temporal graph convolution,thereby enhancing the interpretability and recognition performance.Firstly,the quality of skeleton frames was estimated using the joint confidence to remove low-quality skeleton frames,addressing the skeleton noise problem.Secondly,based on the domain knowledge of human activity,an adaptive temporal dynamic frame selection module was proposed for calculating the motion salient regions to capture the dynamic patterns of key skeleton frames in human motion.To represent the intrinsic topology of human joints,an improved spatiotemporal graph convolutional network was used for interpretable skeleton-based action recognition.Experiments were conducted on three large public datasets,including NTU RGB+D,NTU RGB+D 120,and FineGym,and the results demonstrated that the recognition accuracy of this method outperformed comparative methods and possessed interpretability.