基于多粒度跨模态特征增强的红外与可见光图像融合

Infrared and Visible Image Fusion Based on Multi-granularity Cross-modal Feature Enhancement

针对红外与可见光图像融合中跨模态特征提取与整合不充分的问题,提出了一种基于Transformer和卷积神经网络(CNN)的图像融合算法。为充分提取深层全局上下文特征,设计了以Transformer为主体的深层特征提取模块,Transformer提取的多粒度全局上下文特征被馈送入跨模态特征增强模块(CFEB),CFEB以自上而下的方式充分整合双模态深度特征,整合后的融合特征与双模态特征在通道维度连接,用以重建融合图像。在MSRS公开数据集上的大量定性与定量实验结果表明,所提方法可以充分整合红外与可见光跨模态互补信息,获得显著的图像融合效果。

This work proposes a Transformer and CNN based image fusion algorithm to address the issues of insufficient cross-modal feature integration in infrared and visible image fusion.To fully extract deep features of global context,a deep feature extraction module with Transformer blocks is designed.The multi-granularity global context features extracted by the Transformer blocks are fed into a cross-modal feature enhancement module(CFEB)to fully integrate the dual-modality deep features in a top-down manner.The integrated fused features are connected with the dual-modality features in the channel dimension to reconstruct the fused image.A large number of qualitative and quantitative experimental results on public MSRS dataset show that the proposed method can fully integrate complementary information from infrared and visible images,achieving significant image fusion effects.