基于深度学习的像素级全色图像锐化研究综述

Deep-learning approaches for pixel-level pansharpening

全色图像锐化是遥感数据处理领域的一个基础性问题,在地物分类、目标识别等方面具有重要的研究意义和应用价值。近年来,深度学习在自然语言处理、计算机视觉等领域取得了巨大进展,也推动了像素级全色图像锐化技术的发展。本文提出从经典方式和协同方式两个方面对深度学习在全色图像锐化中的研究进行系统的综述,并在此基础上进行前景展望。首先,给出全色图像锐化常用的数据集和全色图像锐化的质量评价指标;接着,从经典方式与协同方式两个方面对基于深度学习的全色图像锐化最新研究成果进行分门别类的介绍,并进行算法性能的对比、分析和归纳;然后,对全色图像锐化的3个主要应用领域如地物分类、目标识别和地表变化检测进行分析;最后,本文探讨了基于深度学习的全色图像锐化的5个未来研究方向。

Pansharpening is a fundamental problem in the field of remote sensing data processing.It has important research significance and application value in ground object classification and ground surface change detection.In recent years,Deep Learning(DL) has made great progress in natural language processing,computer vision,etc.and has promoted the development of pixel-level pansharpening technology.This work presents a systematic review of the research of DL in pansharpening from two aspects(classical and collaborative approaches) and makes a prospect on this basis.First,the common datasets of pansharpening and the objective evaluation indexes of pansharpening,including reference and non-reference quality evaluation indexes,are provided.Second,the latest research results of DLbased pansharpening are introduced in two different categories from the classical and collaborative methods,and the performance of their algorithms is compared,analyzed,and summarized.The classical methods mainly include AE-based pansharpening,CNN-based pansharpening,DRN-based pansharpening,and GAN-based pansharpening methods.Meanwhile,the collaborative methods mainly include DL+CS-based pansharpening,DL+MRA-based pansharpening,DL+MB-based pansharpening,DL+injection model-based pansharpening,CNN+DRN-based pansharpening,and RNN+CNN-based pansharpening methods.In the comparative analysis of the classical and collaborative methods,the common point is that the DL technology can automatically learn the advantages of complex data features and extract the feature information of the MS or PAN image(i.e.,the information that needs to be retained in the HRMS fusion image).The difference is that the structure of the classical mode is more concise,while that of the collaborative mode is more complex because it is the combination of multiple methods or frameworks.In addition,most early DL-based pansharpening methods utilized the powerful data fitting ability of the DL model and seldom paid attention to the field of pansharpening problems.With the gradual deepening of research,such as using DL methods combined with traditional pansharpening methods,this designed fusion model considers spectral and spatial distortions.Accordingly,the DL methods can further enhance the pansharpening effect.Thirdly,the three main application fields of pansharpening are analyzed,such as object classification,target recognition,and surface change detection.Finally,this work discusses the future research direction of DL-based pansharpening in combination with remote sensing knowledge to fully tap the potential of DL to obtain fused images with richer details and more natural spectra.For example,for the evaluation of pansharpening application,the performance of pansharpening in a certain application is related not only to the high quality of fusion image but also to the knowledge of a specific application field.Accordingly,the application-oriented pansharpening evaluation algorithms will be the focus of future study.Furthermore,DL-based pansharpening needs to train a large number of network parameters,resulting in a longer training time for the pansharpening model.The lightweight depth model has a smaller network capacity,lower time complexity,and lower hardware requirements.Therefore,constructing a lightweight pansharpening model is a promising future direction.