多重关系感知的红外与可见光图像融合网络

Infrared and Visible Image Fusion Network with Multi-Relation Perception

为充分整合红外与可见光图像间的一致特征和互补特征,该文提出一种基于多重关系感知的红外与可见光图像融合方法。该方法首先利用双分支编码器提取源图像特征,然后将提取的源图像特征输入设计的基于多重关系感知的跨模态融合策略,最后利用解码器重建融合特征生成最终的融合图像。该融合策略通过构建特征间关系感知和权重间关系感知,利用不同模态间的共享关系、差分关系和累积关系的相互作用,实现源图像一致特征和互补特征的充分整合,以得到融合特征。为约束网络训练以保留源图像的固有特征,设计了一种基于小波变换的损失函数,以辅助融合过程对源图像低频分量和高频分量的保留。实验结果表明,与目前基于深度学习的图像融合方法相比,该文方法能够充分整合源图像的一致特征和互补特征,能够有效保留可见光图像的背景信息和红外图像的热目标,整体融合效果优于对比方法。

A multi-relation perception network for infrared and visible image fusion is proposed in this paper to fully integrate consistent features and complementary features between infrared and visible images.First,a dual-branch encoder module is used to extract features from the source images.The extracted features are then fed into the fusion strategy module based on multi-relation perception.Finally,a decoder module is used to reconstruct the fused features and generate the final fused image.In this fusion strategy module,the feature relationship perception and the weight relationship perception are constructed by exploring the interactions between the shared relationship,the differential relationship,and the cumulative relationship across different modalities,so as to integrate consistent features and complementary features between different modalities and obtain fused features.To constrain network training and preserve the intrinsic features of the source images,a wavelet transform-based loss function is developed to assist in preserving low-frequency components and highfrequency components of the source images during the fusion process.Experiments indicate that,compared to the state-of-the-art deep learning-based image fusion methods,the proposed method can fully integrate consistent features and complementary features between source images,thereby successfully preserving the background information of visible images and the thermal targets of infrared images.Overall,the fusion performance of the proposed method surpasses that of the compared methods.