基于双分支边缘卷积融合网络的红外与可见光图像融合方法

A Dual Branch Edge Convolution Fusion Network for Infrared and Visible Images

提出一种基于双分支边缘卷积融合网络的红外与可见光图像融合方法。首先,提出一种改进的双分支边缘卷积结构,将图像包含的信息分解为公共信息和模态信息,并于每个分支引入边缘卷积块,更好的提取深度特征;然后在融合层引入卷积注意力模块对模态特征进行增强;最后基于所本文编解码网络特点,提出一种重建损失结合融合损失的损失函数。经过大量的消融性实验和对比实验表明,本文方法能够很好的保留原图像中的公共信息和模态信息,并且相比目前最新的融合方法在主观和客观评价上都具有优秀的综合性能。

Image fusion technology is the process of extracting and integrating complementary information from a s et o f i mages,and fusing them into a single image.This process aggregates more effective information,removes redundant information,and enhances the quality of information and scene perception capabilities in the image.Among them,infrared and visible image fusion is a common branch in the field of image fusion and is widely used in the field of image processing.Infrared images can capture hidden heat source targets and have strong anti-interference capabilities.Visible images have rich scene information through reflective imaging.The fusion of the two can complement the rich detail texture information in the visible image and the highlighted target information in the infrared image,obtain a clearer and more accurate description of the scene content,which is beneficial for target recognition and tracking.However,most of the current fusion methods based on deep learning focus on feature extraction and design of loss function,and do not separate public information from modal information,and use the same feature extractor to extract features of different modes without considering the differences between different modes.Based on this,this paper proposes an infrared and visible image fusion method based on a dual-branch edge convolution fusion network.First,based on the dual-branch autoencoder,an improved dual-branch edge convolution structure is proposed,which decomposes the extracted feature information into common information and modality information,and introduces an edge convolution block in each branch to better extract deep features;then a convolutional block attention module is introduced in the fusion layer to enhance the features of different modalities separately for better fusion effect;finally,based on the characteristics of the encoding and decoding network in this paper,a loss function combining reconstruction loss and fusion loss is proposed,which better retains the information of the source image.In order to verify the effectiveness of the proposed method,10 pairs images were randomly selected on the TNO dataset and the test set of MSRS dataset respectively to test on 6 indicators,s uch a s M SE,SF,CC,PSNR,Q F AB,and MS-SSIM.Firstly,four sets of ablation experiments were designed to verify the effectiveness of the edge convolution block and the convolutional block attention module.The results show that the edge convolution block can more effectively extract the features of the image,retain more edge information,and the fusion effect of the convolutional block attention module on modality information is also significantly enhanced.In addition,the optimal parameters of the loss function are found asα=5.0,μ=1.0 by grid search method.Besides,the proposed method is compared with the mainstream infrared and visible image fusion methods,including SeAFusion,SwinFuse,etc.The results show that the proposed method retains the high-brightness targets of the infrared image and the clear background of the visible image,w ith a higher contrast,and has a better visual effect.To be specific,the proposed method in this paper leads other methods in the four indicators of MSE,CC,PSNR and MS-SSIM,with the best overall quality.The above experimental results prove that compared with other methods,the fusion result of the proposed method can better retain the thermal radiation information of the infrared image and the texture information of the visible image,and surpasses the existing Infrared and Visible Image Fusion methods in terms of comprehensive performance.Although the experiment was only tested on the task of Infrared and Visible Image Fusion,the method in this paper can also be extended to the fusion of more than two modalities.Future work will continue to test its performance in other multi-modal information fusion tasks,and optimize the network structure to obtain better fusion results.