基于多视图自适应3D骨架网络的工业装箱动作识别

Industrial box-packing action recognition based on multi-view adaptive 3D skeleton network

目的 动作识别在工业生产制造中变得越来越重要。但在复杂的生产车间内,动作识别技术会受到环境遮挡、视角变化以及相似动作识别等干扰。基于此,提出一种结合双视图骨架多流网络的装箱行为识别方法。方法 将堆叠的差分图像(residual frames, RF)作为模型的输入,结合多视图模块解决人体被遮挡的问题。在视角转换模块中,将差分人体骨架旋转到最佳的虚拟观察角度,并将转换后的骨架数据传入3层堆叠的长短时记忆网络(long short-term memory, LSTM)中,将不同视角下的分类分数进行融合,得到识别结果。为了解决细微动作的识别问题,采用结合注意力机制的局部定位图像卷积网络,传入到卷积神经网络中进行识别。融合骨架和局部图像识别的结果,预测工人的行为动作。结果 在实际生产环境下的装箱场景中进行了实验,得到装箱行为识别准确率为92.31%,较大幅度领先于现有的主流行为识别方式。此外,该方法在公共数据集NTU(Nanyang Technological University) RGB+D上进行了评估,结果显示在CS(cross-subject)协议和CV(cross-view)协议中的性能分别达到了85.52%和93.64%,优于其他网络,进一步验证了本文方法的有效性和准确性。结论 本文提出了一种人体行为识别方法,能够充分利用多个视图中的人体行为信息,采用骨架网络和卷积神经网络模型相结合的方式,有效提高了行为识别的准确率。

Objective Action recognition has become increasingly important in industrial manufacturing.Production effi⁃ciency and quality can be improved by recognizing worker actions and postures in complex production environments.In recent years,action recognition based on skeletal data has received widespread attention and research,with methods mainly based on graph convolutional networks(GCN)or long short-term memory(LSTM)networks exhibiting excellent recognition performance in experiments.However,these methods have not considered the recognition problems of occlu⁃sion,viewpoint changes,and similar subtle actions in the factory environment,which may have a significant impact on subsequent action recognition. Therefore, this study proposes a packing behavior recognition method that combines a dualview skeleton multi-stream network. Method The network model consists of a main network and a sub-network. The mainnetwork uses two RGB videos from different perspectives as input and records the input of workers at the same time andaction. Subsequently, the image difference method is used to convert the input video data into a difference image. More⁃over, the 3D skeleton information of the character is extracted from the depth map by using the 3D pose estimation algo⁃rithm and then transmitted to the subsequent viewing angle conversion module. In the perspective conversion module, therotation of the bone data is used to find the best viewing angle, and the converted skeleton data are passed into a three-layerstacked LSTM network. The different classification scores of the weighted fusion are obtained for the recognition results ofthe main network. In addition, for some similar behaviors and non-compliant “fake actions”, we use a local positioningimage convolution network combined with an attention mechanism and pass it into the ResNeXt network for recognition.Moreover, we introduce a spatio-temporal attention mechanism for analyzing video action recognition sequences to focus onthe key frames of the skeleton sequence. The recognition scores of the main network and the sub-network are fused in pro⁃portion to obtain the final recognition result and predict the behavior of the person. Result First, convolutional neural net⁃work (CNN)-based methods usually have better performance than recurrent neural network (RNN)-based ones, whereasGCN-based methods have middling performance. Moreover, CNN and RNN network structures are combined to improvethe accuracy and recall rate to greatly explore the spatiotemporal information of skeletons. However, the method proposedin this study has an identification accuracy of packing behavior of 92. 31% and a recall rate of 89. 72%, which is still3. 96% and 3. 81% higher than the accuracy, respectively. The proposed method is significantly ahead of other existingmainstream behavior recognition methods. Second, the method based on a difference image combined with a skeletonextraction algorithm can achieve an 87. 6% accuracy, which is better than RGB as the input method of the original image,although the frame rate is reduced to 55. 3 frames per second, which is still within the acceptable range. Third, consider⁃ing the influence of the adaptive transformation module and the multi-view module on the experiment, we find that the rec⁃ognition rate of the single-stream network with the adaptive transformation module is greatly improved, but the fps isslightly decreased. The experiment finds that the learning of the module is more inclined to observe the action from the frontbecause the front observation can scatter the skeleton as much as possible compared with the side observation. The highestdegree of mutual occlusion among bones was the worst observation effect. For dual view, simply fusing two different singlestream output results can improve the performance, and the weighted average method has the best effect, which is 3. 83%and 3. 03% higher than the accuracy of single-stream S1 and S2, respectively. Some actions have the problem of objectocclusion and human self-occlusion under a certain shooting angle. The occlusion problem can be solved by two comple⁃mentary views, that is, the occluded action can be well recognized in one of the views. In addition, evaluations were car⁃ried out on the public NTU RGB+D dataset, where the performance results outperformed other networks. This result furthervalidates the effectiveness and accuracy of the proposed method in the study. Conclusion This method uses a two-streamnetwork model. The main network is an adaptive multi-view RNN network. Two depth cameras under complementary per⁃spectives are used to collect the data from the same station, and the incoming RGB image is converted into a differentialimage for extracting skeleton information. Then, the skeleton data are passed into the adaptive view transformation moduleto obtain the best skeleton observation points, and the three-layer stacked LSTM network is used to obtain the recognitionresults. Finally, the weighted fusion of the two view features is used, and the main network solves the influence of occlu⁃sion and background clutter. The sub-network adds the hand image recognition of skeleton positioning, and the interceptedlocal positioning image is sent to the ResNeXt network for recognition to make up for the problem of insufficient accuracy of“fake action” and similar action recognition. Finally, the recognition results of the main network and the sub-network arefused. The human behavior recognition method proposed in this study effectively utilizes human behavior information frommultiple views and combines skeleton network and CNN models to significantly improve the accuracy of behavior recogni⁃tion.