基于多层记忆增强和残差时空变换器的鱼类异常运动行为检测

Abnormal fish movement behavior detection based on multilayer memory enhancement and residual spatio-temporal transformer

由于人工提取抽象特征方法捕获视频异常存在特征学习不足、特征选择困难和泛化性差的问题,笔者将计算机视觉技术引入鱼群异常运动行为检测研究中,采用无监督的学习方式,提出一种结合多层记忆增强和残差时空变换器的鱼类异常运动行为检测方法。该方法以U-Net网络为基础,利用其编码器和解码器对视频帧编码和解码,并根据预测帧和真实帧之间的差异实现异常行为检测。为了加强连续视频帧之间的时空信息特征联系,提出残差时间变换器模块和残差空间变换器模块以提升网络对时间信息和空间信息的建模能力。由于卷积神经网络具有一定的泛化能力,使用记忆增强模块代替U-Net网络中的跳跃连接,降低编码器对异常帧的表示能力。此外,采用生成对抗网络(GAN)技术生成更加真实的预测帧,从而提升网络的检测精度。结果表明:该方法能有效提取鱼群的运动特性和外观特性,在自制的两类鱼群数据集上的AUC(曲线下面积)分别达0.916和0.921,实现了鱼群异常运动行为检测。

features to capture video anomalies was no longer applicable to large-scale aquaculture due to the problems of insufficient feature learning,difficult feature selection and poor generalization.In this study,computer vision technology was introduced into the study of fish movement behavior anomaly detection,and an unsupervised learning approach was used to propose a fish movement behavior anomaly detection method that combined multilayer memory enhancement and residual spatio-temporal transformer to effectively extract the motion correlation and appearance characteristics of fish.Firstly,based on U-Net network,its encoder and decoder were used to implement encoding and decoding of video frames,and behavior anomaly detection was achieved based on the difference between predicted and real frames.In order to strengthen the connection of spatio-temporal information features between consecutive video frames,the residual temporal transformer module and the residual spatial transformer module were proposed to enhance the network’s ability to model temporal and spatial information.Since the convolutional neural network had certain generalization ability,the memory enhancement module was used instead of the jump connection in the U-Net network to alleviate the ability of the encoder to represent the anomalous frames.In addition,Generative Adversarial Networks was used to generate more realistic prediction frames,thereby improving the detection accuracy of the network.The results indicated that this method could effectively extract the motion and appearance characteristics of fish.On two self-made fish datasets,the area under the curve(AUC)reached 0.916 and 0.921,respectively,achieving fish movement behavior anomaly detection.