信息分离和质量引导的红外与可见光图像融合

Information decomposition and quality guided infrared and visible image fusion

目的 红外与可见光图像融合的目标是将红外图像与可见光图像的互补信息进行融合,增强源图像中的细节场景信息。然而现有的深度学习方法通常人为定义源图像中需要保留的特征,降低了热目标在融合图像中的显著性。此外,特征的多样性和难解释性限制了融合规则的发展,现有的融合规则难以对源图像的特征进行充分保留。针对这两个问题,本文提出了一种基于特有信息分离和质量引导的红外与可见光图像融合算法。方法 本文提出了基于特有信息分离和质量引导融合策略的红外与可见光图像融合算法。设计基于神经网络的特有信息分离以将源图像客观地分解为共有信息和特有信息,对分解出的两部分分别使用特定的融合策略;设计权重编码器以学习质量引导的融合策略,将衡量融合图像质量的指标应用于提升融合策略的性能,权重编码器依据提取的特有信息生成对应权重。结果 实验在公开数据集RoadScene上与6种领先的红外与可见光图像融合算法进行了对比。此外,基于质量引导的融合策略也与4种常见的融合策略进行了比较。定性结果表明,本文算法使融合图像具备更显著的热目标、更丰富的场景信息和更多的信息量。在熵、标准差、差异相关和、互信息及相关系数等指标上,相较于对比算法中的最优结果分别提升了0.508%、7.347%、14.849%、9.927%和1.281%。结论 与具有领先水平的红外与可见光算法以及现有的融合策略相比,本文融合算法基于特有信息分离和质量引导,融合结果具有更丰富的场景信息、更强的对比度,视觉效果更符合人眼的视觉特征。

Objective Infrared and visible image fusion is essential to computer vision and image processing. To strengthen the scenes recognition derived of multisource images, more multi-sensors imagery information is required to be fused in relation to infrared and visible images. A fused image is generated for human perception-oriented visual tasks like video surveillance, target recognition and scene understanding. However, the existing fusion methods are usually designed by manually selecting the characteristics to be preserved. The existing fusion methods can be roughly divided into two categories in the context of traditional fusion methods and the deep learning-based fusion methods. For the traditional methods, to comprehensively characterize and decompose the source images, they need to manually design transformation methods. The fusion strategies are manually designed to fuse the decomposed subparts. The manually designed decomposition methods become more and more complex, which leads to the decline of fusion efficiency. For the deep learning-based methods, some methods define the unique characteristics of source images via human observation. The fused images are expected to preserve these characteristics as much as possible. However, it is difficult and unsuitable to identify the vital information through one or a few characteristics. Other methods are focused on preserving higher structural similarity with source images in terms of the fused image. It will reduce the saliency of thermal targets in the fusion result, which is not conductive to the rapid location and capture of thermal targets by the human vision system. Our method is designed to solve these two issues. We develop a new deep learning-based decomposition method for infrared and visible image fusion. Besides, we propose a deep learning-based and quality-guided fusion strategy to fuse the decomposed parts. Method Our infrared and visible image fusion method is based on the information decomposed and quality-guided fusion strategy. First, we design an image decomposition and representation way, which is based on the convolution neural network(CNN). For each source image, two encoders are used to decompose the source images into the common part and unique part. Based on three loss functions(including a reconstruction loss, a translation loss and a loss to constrain the unique information), four encoders are learned to realize the physical-meaning-related decomposition. For the two decomposed parts, specific fusion strategies are applied for each. For the common parts, a traditional fusion strategy is selected to reduce the computational complexity and improve the fusion efficiency. For the unique parts, a weight encoder is assigned to learn the quality-guided fusion strategy, which can further preserve the complementary information derived from multi-source images. To improve the fusion performance, the metrics is used to evaluate the quality of fused images. The weight encoder generates the corresponding weights according to the unique information. The generator-optimized in the unique information decomposition procedure is used to generate the final fused image according to the fused common part and unique part. Result Our method is compared to six state-of-the-art visible and infrared image fusion methods on the publicly available dataset named as RoadScene. In addition, the quality-guided fusion strategy is also in compared with four common fusion strategies, including mean, max, addition and l1-norm on the publicly available dataset. The qualitative comparisons show that our fusion results have three priorities as mentioned below: first, our fusion results can highlight thermal targets. It is beneficial to capture the thermal targets in accordance with the high contrast. Second, more scene information and clearer edges or textures can be presented. Some regions and textures are enhanced as well. Third, even in some extreme cases, our fusion results also show the most information. The effective information in one source image is preserved in the fused image without being affected by the regions in the other source image which has less information. Additionally, we also perform the quantitative evaluation of the proposed method with comparative fusion methods and strategies on six objective metrics. These metrics is composed of entropy, standard deviation, the sum of difference correlation, mutual information and correlation coefficient. Our method shows the best or comparable performance. Compared to existing fusion methods, our average is increased by 0.508%, 7.347%, 14.849%, 9.927% and 1.281% compared with existing methods, respectively. Furthermore, our method is applied to fuse a RGB visible image and a single-channel infrared image. The results show that our method is feasible to improve fusion results. Conclusion We develop an infrared and visible image fusion method based on information decomposition and quality-guided fusion strategy. The experiment results show that the proposed fusion method and fusion strategy outperforms several state-of-the-art infrared and visible image fusion methods and the existing fusion strategies. Both the qualitative and quantitative results show the effectiveness of the proposed method and strategy.