基于条件生成对抗网络的手语样本骨架缺失关节点修复

Missing Joint Point Repair of Sign Language Sample Skeleton Based on Conditional Generation Adversarial Networks

计算机视觉技术由于受到遮挡、视角和光照等因素的影响,对手语样本骨架关节点的检测通常存在缺失,导致手语识别准确率降低.为此,提出基于条件生成对抗网络(CGAN)的手语样本骨架缺失关节点修复方法.首先,通过分析手语样本残缺骨架中关节点的缺失分布情况,构建缺失关节点分布概率模型;其次,对完整骨架引入分布概率模型生成的缺失关节点,将这些残缺骨架用于CGAN框架中生成器和判别器的训练,通过CGAN框架训练好的生成器能够以残缺骨架为条件生成没有缺失的骨架;最后,用生成骨架去补全残缺骨架,即完成了修复.在中国手语数据集CSL上开展实验,生成器迭代训练80次后,生成骨架与完整骨架的平均均方根误差从0.019减小到0.001;在修复骨架缺失关节点的手语样本上,搭建手语识别网络迭代训练120次,与未进行修复相比,其识别准确率从90.6%提升为99.6%.实验结果表明,该方法能够有效地修复缺失关节点,极大地提升手语识别准确率.

Due to the influence of occlusion,angle of view,illumination and other factors,the detection of skeleton joint points of sign language samples by computer vision technology is usually missing,which leads to the lower accuracy of sign language recognition.To solve this problem,a new method based on conditional generation adversarial networks(CGAN)was proposed to repair the missing joint points of sign language sample skeleton.Firstly,by analyzing the distribution of missing joint points in the incomplete skeleton of sign language samples,the distribution probability model of missing joint points was constructed.Secondly,the missing joint points generated by the distributed probability model are added into the complete skeleton,and uses these incomplete skeletons for the training of the generator and discriminator in the CGAN framework,the generator trained through the CGAN framework can generate skeletons without missing joint points on the condition of the incomplete skeleton.Finally,we use the generated skeleton to fill the incomplete skeleton and complete the repair.Experiments are carried out on the Chinese sign language dataset CSL,after the generator iteratively trained for 80 times,the average root mean square error between the generated skeleton and the complete skeleton is reduced from 0.019 to 0.001.120 times of iterative training of sign language recognition network was built on the sign language samples with missing joint points repaired,compared with no repair,the recognition accuracy is increased from 90.6%to 99.6%.The experimental results show that the proposed method can effectively repair the missing joints and greatly improve the accuracy of sign language recognition.