基于声音与视觉特征多级融合的鱼类行为识别模型U-FusionNet-ResNet50+SENet

A fish behavior recognition model based on multi-level fusion of sound and vision U-fusionNet-ResNet50+SENet

为解决在光线昏暗、声音与视觉噪声干扰等复杂条件下,单模态鱼类行为识别准确率和召回率低的问题,提出了基于声音和视觉特征多级融合的鱼类行为识别模型U-FusionNet-ResNet50+SENet,该方法采用ResNet50模型提取视觉模态特征,通过MFCC+RestNet50模型提取声音模态特征,并在此基础上设计一种U型融合架构,使不同维度的鱼类视觉和声音特征充分交互,在特征提取的各阶段实现特征融合,最后引入SENet构成关注通道信息特征融合网络,并通过对比试验,采用多模态鱼类行为的合成加噪试验数据验证算法的有效性。结果表明:U-FusionNet-ResNet50+SENet对鱼类行为识别准确率达到93.71%,F1值达到93.43%,召回率达到92.56%,与效果较好的已有模型Intermediate-feature-level deep model相比,召回率、F1值和准确率分别提升了2.35%、3.45%和3.48%。研究表明,所提出的U-FusionNet-ResNet50+SENet识别方法,可有效解决单模态鱼类行为识别准确率低的问题,提升了鱼类行为识别的整体效果,可以有效识别复杂条件下鱼类的游泳、摄食等行为,为真实生产条件下的鱼类行为识别研究提供了新思路和新方法。

In order to solve the problem of low accuracy and recall rate of single-mode fish behavior recognition under complex conditions such as dim light,sound and visual noise interference,a multi-level integration of sound and visual features of fish behavior recognition model U-FusionNet-ResNet50+SENet was proposed by ResNet50 model to extract visual modal features.Sound modal characteristics were extracted by MFCC+RestNet50 model.On this basis,a U-shaped fusion architecture was designed to fully interact the visual and sound features of fish behaviors with different dimensions,and to realize feature fusion in each stage of feature extraction.Finally,SENet was introduced to form a feature fusion network of attention channel information,and the effectiveness of the algorithm was verified by the synthetic test data of multi-modal fish behaviors through comparative experiments.The results showed that the accuracy rate of fish behavior recognition by U-FusionNet-ResNet50+SENet reached 93.71%,F1 score 93.43%and recall rate 92.56%.Compared with the existing Intermediate-feature-level deep model with better effect,there was increase in recall rate by 2.35%,F1 value by,3.45%and accuracy by 3.48%,indicating that the U-FusionNet-ResNet50+SENet recognition method proposed in this paper can effectively solve the problem of low accuracy of single-mode fish behavior recognition,and improve the overall effect of fish behavior recognition.