基于频谱特征混合Transformer的红外和可见光图像融合

Infrared and visible image fusion based on spectral feature hybrid Transformer

为了解决传统红外与可见光图像融合方法对细节与频率信息表征能力不足、融合结果存在模糊伪影的问题,提出一种基于频谱特征混合Transformer的红外和可见光图像融合算法。在Transformer的基础上,利用傅里叶变换将图像域特征映射到频域,设计了一种新的复数Transformer来提取源图像的深层频域信息,并与图像域特征进行混合,以此提高网络对细节与频率信息的表征能力。此外,在图像重建前设计了一种新的令牌替换模块,动态评估Transformer令牌的显著性后消除得分较低的令牌,防止融合图像出现伪影。在MSRS数据集上进行的定性和定量实验结果显示,与九种先进的算法相比,该算法具有较好的融合效果。

Aiming at the problem of insufficient representation capability for details and frequency information in traditional infrared and visible image fusion methods,this paper proposed a fusion algorithm based on spectral feature hybrid Transformer.The algorithm utilized the Fourier transform to map image domain features to the frequency domain.Then the novel complex Transformer extracted deep-frequency information from the source images and mixed them with the features of the image domain to enhance the representation capability for edges and details.Additionally,the algorithm used a token replacement module to evaluate the saliency of Transformer tokens and eliminate the tokens with lower scores to prevent the presence of artifacts in the fused image.Qualitative and quantitative experiments conducted on the MSRS dataset demonstrate that the proposed algorithm exhibits superior fusion performance compared to nine state-of-the-art methods.