基于CNN-Transformer的自编码器红外和可见光图像融合方法

Infrared and Visible Image Fusion Based on Autoencoder Composed of CNN-transformer

基于自编码器的图像融合模型因无需手动设计融合规则而受到更多关注。然而,该融合网络编码器采用的卷积神经网络仅对局部感受野敏感,无法提取图像全局特征,且缺乏从红外图像和可见光图像中提取独特特征的能力。本文构建了一种基于自动编码器的新型图像融合网络,该网络由编码器模块、融合模块和解码器模块组成。在编码器模块中,结合使用CNN和Transformer模块以同时捕捉原图像的局部和全局特征。此外,为提取原图像特定信息,分别为原红外和可见光图像设计对比度增强和梯度增强模块。编码器模块获得的特征图像经融合模块串联后输入解码器模块,从而获得融合图像。在三个数据集上的实验结果表明,本文提出的融合网络能较好地保留了红外图像和可见光图像的清晰目标和细节信息,在主观和客观评价方面均优于其他先进方法。同时,本文提出的网络所获得的融合图像在目标检测中获得了最高的平均精度,证明图像融合有利于下游任务。

Image fusion model based on autoencoder network gets more attention because it does not need to design fusion rules manually.However,most autoencoder-based fusion networks use two-stream CNNs with the same structure as the encoder,which are unable to extract global features due to the local receptive field of convolutional operations and lack the ability to extract unique features from infrared and visible images.A novel autoencoder-based image fusion network which consist of encoder module,fusion module and decoder module is constructed in this paper.In the encoder module,the CNN and Transformer are combined to capture the local and global feature of the source images simultaneously.In addition,novel contrast and gradient enhancement feature extraction blocks are designed respectively for infrared and visible images to maintain the information specific to each source images.The feature images obtained by encoder module are concatenated by the fusion module and input to the decoder module to obtain the fused image.Experimental results on three datasets show that the proposed network can better preserve both the clear target and detailed information of infrared and visible images respectively,and outperforms some state-of-the-art methods in both subjective and objective evaluation.Meanwhile,the fused image obtained by the proposed network can acquire the highest mean average precision in the target detection which proves that image fusion is beneficial for downstream tasks.