CNN A-BLSTM network的双人交互行为识别

CNN A-BLSTM network for two-person interaction behavior recognition

关节点数据结合卷积神经网络用于双人交互行为识别存在图像化过程中对交互信息表达不充分且不能有效建模时序关系问题,而结合循环神经网络中存在侧重于对时间信息的表示却忽略了双人交互空间结构信息构建的问题。为此提出一种新的卷积神经网络结合加入注意机制的双向长短时期记忆网络(CNN A-BLSTM)模型。首先对每个人的关节点采用基于遍历树结构进行排列,然后对视频中的每一帧数据构建交互矩阵,矩阵的中的数值为排列后双人之间所有的关节点坐标间的欧氏距离,将矩阵进行灰度图像编码后所得图像依次送入CNN中提取深层次特征得到特征序列,然后将所得序列送入A-BLSTM网络中进行时序建模,最后送入Softmax分类器得到识别结果。将新模型用于NTU RGB D数据集中的11类双人交互行为的识别,其准确率为90%,高于目前的双人交互行为识别算法,验证了该模型的有效性和良好的泛化性能。

Joint data combined with convolutional neural network for two-person interaction behavior recognition has the problem of insufficient expression of interactive information during the imaging process and ineffective modeling of time-series relations.In combination with recurrent neural network,there is a problem that focuses on the representation of time information.However,it ignores the problem of constructing information about the spatial structure of the two-person interaction.Therefore,a novel model named CNN attention-bidirectional long short-term memory(CNN A-BLSTM)network is proposed.First,the joints of each person are arranged based on the traversal tree structure,and then the interaction matrix is constructed for each frame of data in the video.The values in the matrix are the Euclidean distance between the arranged joint coordinates of two persons.After encoding the gray-scale image of the matrix,the images are sequentially sent to CNN to extract deep-level features to obtain the feature sequence.And then the obtained feature sequence is sent to the A-BLSTM network for time series modeling,and finally sent to the Softmax classifier to obtain the recognition result.The new model is applied to 11 types of two-person interaction in NTU RGB D dataset,and the accuracy is 90%,which is higher than the current two-person interaction recognition algorithm.The effectiveness and good generalization performance of the new model are verified.