双判别器深度残差GAN高光谱图像融合

Two-discriminators-deep residual GAN hyperspectral image pan-sharpening

目的 为了解决高空间分辨率多光谱图像与高光谱图像融合时的多波段对多波段问题,以及高空间分辨率多光谱图像波谱范围不能完全涵盖高光谱图像波谱范围而导致的光谱失真问题,本文利用深度学习的数据驱动优势,基于高分5号(GF-5)高光谱数据和Sentinel-2多光谱数据,提出一种基于生成对抗网络(generative adversarial network,GAN)的高光谱图像空谱融合方法——双判别器深度残差GAN网络(two discriminator deep residual GAN,2DDRGAN)。方法 考虑待融合图像间的波谱范围关系,采用分组融合策略,利用波段间的相关性,将多对多的融合问题转变为多个一对多的融合问题。使用深度残差模块深度提取图像的光谱和空间特征,用两个判别网络对融合图像的空间和光谱质量分别进行判断,改善生成网络生成的融合图像质量。另外,本文的深度学习网络不需要制作额外的融合结果标签,待融合图像本身就是标签,这大大降低了高光谱融合的工作量,也是目前深度学习遥感图像融合的根本改变。结果 与常用传统空谱融合方法和经典深度学习方法比较的实验结果表明,对于不同地物类型数据,该网络得到的融合结果在提升空间分辨率的同时,有较高的光谱保真度。光谱曲线评价也验证了该网络对于高空间分辨率图像波谱范围以外的高光谱图像波段进行融合时有良好的光谱保真度。结论 本文方法通过深度残差模块提取高光谱图像光谱特征和高空间分辨率图像空间特征,同时引入双判别网络,使得融合结果在保持光谱信息的同时更好地提升空间信息。

Objective Hyperspectral image(HS) pan-sharpening can obtain the fused image with both high spatial resolution and high spectral resolution by using the complementary information between the high spectral resolution HS and the high spatial resolution multi-spectral image(MS).However,HS and MS images both have multi-bands,which is different from the traditional MS pan-sharpening where one band panchromatic image(PAN) with multi-band MS image.This manyto-many band relationship is not convenient for the implementation of pan-sharpening method.Moreover,for those bands where HS exceeds the spectral range of MS,there will be obvious spectral distortion in the fused image due to the lack of strict physical complementary information.In order to solve the above problems,this paper exploits the data-driven advantages of deep learning and proposes the two discriminators deep residual generative adversarial network(2DDRGAN) for hyperspectral image pan-sharpening based on Gaofen-5(GF-5) HS image and Sentinel-2 MS image.Method Considering the spectral range relationship between HS and MS,this paper adopts the grouping pan-sharpening strategy to transform the many-to-many problem into multiple one-to-many problems by using the correlation between bands of HS and MS.For the HS band within the spectral range of the high spatial resolution MS image,the band of HS image is directly assigned to the group corresponding to the high spatial resolution MS image.For the HS bands outside the spectral range,the correlation coefficient is used as the band grouping standard to group the bands of HS and MS.It solves the problem that two multiband images are not easy to be fused directly,and indirectly improves the spectral fidelity of the fused image.The proposed 2DDRGAN consists of one generation network and two discrimination networks.The generation network mainly extracts the deep spectral and spatial features by establishing the deep residual module.Two discrimination networks judge the spatial quality and the spectral quality of the fused image,respectively,to improve the quality of the output fused image from the generation network.The main task of the spatial discrimination network is to compare the results of the generated network with the high spatial resolution MS image,so as to ensure that the generated fused images have high spatial resolution.The main task of spectral discrimination network is to compare the results of the generated network with the HS images,to ensure the generated fused images have high spectral resolution.Moreover,the deep learning pan-sharpening methods do not have real high spatial resolution and high spectral resolution images as the fusion result labels.Most of them currently are based on the simulated data made by Wald protocol.In this paper,the deep learning network does not need to create additional fusion result labels.The images to be fused themselves are labels,which greatly reduces the workload of hyperspectral fusion with a huge amount of data,and is also a fundamental change in the current deep learning fusion.Result The fusion results of the proposed 2DDRGAN method in different scenarios are compared with traditional methods and existing deep learning methods.The experimental results show that the 2DDRGAN method has high spectral fidelity while improving the spatial resolution.The spectral curve evaluation also verifies shows that the 2DDRGAN network has good spectral fidelity for hyperspectral image bands beyond the spectral range of high spatial resolution images.Conclusion This method extracts the spectral features of hyperspectral images and the spatial features of high spatial resolution images through the depth residual module,and introduces the double discrimination network,so that the fusion results can better improve the spatial information while maintaining the spectral information.