基于多模态特征及卷积神经网络的智慧教室人物行为识别方法

Character behavior recognition method in smart classroom based on multimodal features and convolutional neural network

为了精准识别不同环境多类别人物行为,使其适用于多个场景,研究基于多模态特征及卷积神经网络的智慧教室人物行为识别方法。通过智慧教室多类别人物视频的均匀稀疏采样,获取智慧教室多类别人物RGB图像及人物行为图像,采用由改进3D CNN和引入时空注意力机制的LSTM模型构成C3DP-LA网络,提取多类别人物RGB图像时空特征,同时提取人物行为图像的人物光流特征、重心特征以及三维SIFT特征,组建人物行为组合特征,经基于全连接层的多模态特征融合网络融合人物时空特征和人物行为组合特征后,将多模态融合结果输入softmax分类器,完成智慧教室人物行为识别。实验结果表明:该方法可有效识别智慧教室多类别人物,降低背景、环境因素对多类别人物特征提取的影响,能够精准识别人物动作并准确分类,具有良好的应用性。

In order to accurately identify multi category human behavior in different environments and make it applicable to multiple scenes,a human behavior recognition method based on multi-modal features and convolutional neural network in smart classrooms is studied.Through uniform sparse sampling of multi category person videos in the smart classroom,RGB images and behavior images of multi category people in the smart classroom are obtained.The C3DP-LA network composed of improved 3D CNN and LSTM model with spatiotemporal attention mechanism is used to extract the spatiotemporal characteristics of multi category person RGB images,and the optical flow characteristics,center of gravity characteristics and three-dimensional SIFT characteristics of the person behavior images are extracted to form the character behavior combination features.After the multi-mode feature fusion network based on the full connection layer fuses the time-space characteristics and the combined characteristics of human behavior,the multi-mode fusion results are input into the softmax classifier to complete the human behavior recognition in the smart classroom.The experimental results show that this method can effectively recognize multi class characters in the smart classroom,reduce the influence of background and environmental factors on the feature extraction of multi class characters,and can accurately recognize and classify the actions of people,with good applicability.