基于Transformer的复合材料多源图像实例分割网络

Transformer-based multi-source images instance segmentation network for composite materials

为提高复合材料铺放质量,辅助现场人员快速对缺陷进行检测,提出一种基于Transformer的复合材料多源图像实时实例分割网络Trans-Yolact,用来对复合材料缺陷进行检测、分类、分割。在Yolact网络框架基础上,针对复合材料缺陷特点,从空间域与通道域两个维度,增强网络对复合材料缺陷的检测能力。在空间域上,常规卷积核具有空间尺度的局限性,对狭长形、大尺寸缺陷的检测效果不佳。因此,采用CNN+Transformer架构的BoTNet作为基础主干网络;同时将Transformer引入Yolact网络的FPN结构中,增强网络从非局部空间中获取信息的能力。在通道域上,采用红外与可见光联立的检测方式,并改进主干网络浅层结构,将其分为可见光通道、红外通道、混合通道,混合通道中引入通道域注意力机制,进一步增强网络对红外与可见光图像的综合判断能力。实验结果表明:改进后Trans-Yolact对复合材料缺陷mAP为88.0%,较基准Yolact网络提高5.5%,缺丝、扭转等狭长形缺陷AP提高15.2%、5.1%,包含部分大尺寸缺陷的异物类缺陷AP提高9.1%。最终对Trans-Yolact网络进行结构化剪枝,剪枝后较基准Yolact网络减少26.5%的计算量(FLOPs)、减少44.7%的参数量;检测帧数提高58%,达到57.67 fps。并在大型龙门复合材料自动铺放设备上进行在线测试,可以满足生产过程最大铺放速度1.2 m/s下,复合材料缺陷的实时检测分割。

In order to improve the quality of automatic fiber placement and assist on-site personnel to quickly detect defects, this paper proposes a real-time instance segmentation network named Trans-Yolact, which is based on Transformer. The Trans-Yolact is used to detect, classify and segment multi-spectrum images of composite material defects. Based on Yolact, aiming at the characteristics of composite material defects, Trans-Yolact’s detection ability of composite material defects is enhanced from the two dimensions of space domain and channel domain. In the spatial domain, the convolution kernels have the limitation of spatial scale. The detection of narrow, long, large-size defects is not effective. Therefore, this paper adopts the BoTNet of the CNN+Transformer architecture as backbone;at the same time, the Transformer is introduced into the FPN structure of the Yolact network to enhance the network’s ability to obtain information from non-local spaces. In the channel domain, the infrared and visible simultaneous detection method is adopted, and the shallow structure of the backbone is improved, which is divided into visible channel, infrared channel, and mixed channel. Channel domain attention mechanism is introduced in mixed channel. Enhance the comprehensive judgment ability of the network for infrared and visible images. The results show that the mAP of Trans-Yolact for composite defect detection is 88.0%, which is 5.5% higher than Yolact network, and the AP of narrow defects such as miss and twist are increased by 15.2% and 5.1%. The AP of foreign defects including some large-scale defects is increased by 9.1%. Finally, the Trans-Yolact network is pruned. After pruning, the amount of floating-point operations per second(FLOPs) and parameters are reduced by 26.5% and 44.7% compared with Yolact network. The number of detection frames is increased by 58%, reaching 57.67 fps. And the online test is carried out on the large-scale gantry composite material automatic laying equipment, which can meet the real-time detection and segmentation of composite material defects under the maximum laying speed of 1.2 m/s in the production process.