联合多流融合和多尺度学习的卷积神经网络遥感图像融合方法

Multi-stream Architecture and Multi-scale Convolutional Neural Network for Remote Sensing Image Fusion

为尽可能保持原始低分辨率多光谱(LRMS)图像光谱信息的同时,显著提高融合后的多光谱图像的空间分辨率,该文提出一种联合多流融合和多尺度学习的卷积神经网络遥感图融合方法。首先将原始MS图像输入频谱特征提取子网得到其光谱特征,然后分别将通过梯度算子处理全色图像得到的梯度信息和通过卷积后的全色图像与得到的光谱特征图在通道上拼接输入到具有多流融合架构的金字塔模块进行图像重构。金字塔模块由多个骨干网络组成,可以在不同的空间感受野下进行特征提取,能够多尺度学习图像信息。最后,构建空间光谱预测子网融合金字塔模块输出的高级特征和网络前端的低级特征得到具有高空间分辨率的MS图像。结合WorldView-3卫星获取的图像进行实验,结果表明,所提方法生成的融合图像在主观目视检验和客观评价指标上都优于大多先进的遥感图像融合方法。

In order to make the fused multispectral images preserve the spectral information of the original Low-Resolution Multi-Spectral(LRMS)images as much as possible,and improve the spatial resolution effectively,a new pan-sharpening method based on multi-stream architecture and multi-scale is proposed.Firstly,This paper inputs the original MS image into the spectral feature extraction subnet to obtain its spectral features,and extracts the multi-directional gradient information and spatial structure information from the panchromatic images by the gradient operator and the convolution kernel.Then the extracted feature is added into the pyramid module with multi-stream fusion architecture for image reconstruction.The pyramid module is composed of multiple backbone networks,which can perform feature extraction under different spatial receptive fields,and can learn image information at multiple scales.Finally,a spatial spectrum prediction subnet is constructed to fuse the high-level features output by the pyramid module and the low-level features of the network front end to obtain multispectral images with high spatial resolution.Experiments on images obtained by WorldView-3 satellites show that the fusion images generated by the proposed method are superior to the most of advanced remote sensing image pan-sharpening methods in both subjective visual and objective evaluation indicators.