基于注意力机制的可见光与红外图像融合算法研究

An improved fusion algorithm for visible and infrared image based on attention mechanism

传统的红外与可见光图像融合算法针对不同源图像不具备通用性,基于卷积神经网络(Convolutional neural network,CNN)深度学习算法改善了这一状况,它们主要是通过融合局部特征来进行图像融合,但未考虑图像中存在的长期依赖关系。为了解决上述问题,在对原始数据集扩增基础上,提出并设计了基于注意力机制可见光与红外图像融合模型,弥补了CNN模型不能提取全局上、下文信息的缺陷。融合模型的学习是通过一种新的两阶段训练策略来完成。首先,训练了一个自编码器用来提取多尺度下的深度特征;然后,构建了由卷积神经网络ResNet34和Transformer组成的融合模块,成功地捕获局部和全局特征。本算法在三个公共基准数据集上的实验表明,不仅融合生成高质量图像并有效解决了算法繁杂问题,还提高了源图像对的空间分辨率和光谱保真度,这有利于融合目标跟踪的准确判断。

Although the traditional fusion methods have achieved good fusion performance,their fusion rules are not universal for different source images.CNN-based deep learning methods perform image fusion by fusing local features,but long-range dependencies are not considered in the methods.In order to solve the above problems,transformer-based models are designed to overcome the problem by modeling the long-range dependencies with the help of self-attention mechanism based on the amplification of the original datasets.The proposed fusion model follows a two-stage training approach.An auto-encoder is trained to extract deep features at multiple scales;then,the fusion blocks are comprised of a ResNet34 and a Transformer branch which capture local and long-range features,respectively.Extensive experiments on three benchmark datasets show that the proposed method performs better than many competitive fusion algorithms.Previous complex algorithms are effectively simplified.The proposed algorithm can also improve images spatial resolution and spectral preservation,and it is conducive to the accurate judgment of the fusion target tracking.