利用隐扩散模型的光学遥感影像去云方法

Method for Cloud Removal from Optical Remote-Sensing Image Based on Latent Diffusion Model

云层遮盖会导致影像中的地表信息缺失,严重降低遥感数据的可用性。针对该问题,提出一种利用隐扩散模型的光学遥感影像去云方法。将本文方法在STGAN数据集和SEN12MS-CR Winter数据集进行评估,得到:本文方法在STGAN数据集上的平均信噪比和平均结构相似性指数分别为26.706和0.759,较对比方法的平均信噪比和平均结构相似性指数分别平均提高9.855、0.171;本文方法在SEN12MS-CR Winter数据集上的平均信噪比和平均结构相似性指数分别为28.779和0.798,较对比方法的平均信噪比和平均结构相似性指数分别平均提高7.683、0.124。实验结果表明:本文方法优于对比方法,能较好地实现光学遥感影像去云任务。

Objective Optical remote-sensing images are widely used in land planning,natural-resource monitoring,disaster response,and other fields owing to their timeliness,large observation range,and clear visual characteristics.However,approximately 70%of optical remote-sensing images are occluded by clouds.Cloud occlusion complicates groundinformation extraction,thus severely limiting the application of optical remote-sensing images.Therefore,cloud removal from optical remote-sensing images is necessary in the preprocessing of remote-sensing images.Compared with the conventional method of removing clouds from optical remote-sensing images,cloud removal based on deep learning presents better effect and a higher accuracy,thus mitigating the issues of conventional algorithms.Recently,denoising diffusion probabilistic models(DDPMs)have attracted much attention due to their generation capabilities beyond generative adversarial networks.DDPMs are generative models that can generate high-quality images closely reflecting the distribution of training data and have achieved the best results in terms of image generation,super-resolution,segmentation,and repair.However,they require significant computing resources to perform denoising.By contrast,the latent diffusion model can obtain high-quality images under less demanding computing requirements.Therefore,this study proposes a cloud-removal method based on the latent diffusion model to remove cloud occlusion from optical remotesensing images and restore their surface information.Methods The cloud-removal method using the hidden diffusion model for optical remote-sensing images proposed herein is outlined as follows:first,a perceptual compression model is used to learn a hidden space on a cloudless remote-sensing image,and a hidden-space perception equivalent to the original pixel space is established.Training the DDPM in a hidden space can reduce the computing requirements and ensure high-quality image generation.Subsequently,a cloudy image is added to the hidden space to guide the diffusion model to generate a cloudless image,and noise estimation is performed using a U-Net-like cross-covariance self-attention noise estimation network(NEUTViT).The NEUTViT includes a jump connection,cross-covariance attention mechanism,and gated linear unit,which can effectively utilize low-level features,significantly reduce the computational burden,improve the nonlinear characterization ability,and achieve more accurate noise estimation.Additionally,the loss of a similar structural constraint is introduced in the forward process to alleviate the randomness of model generation and guide the model to generate cloudless images closer to the source image,thereby achieving a better cloud-removal effect.Results and Discussions First,the possibility of applying the latent diffusion model for removing clouds in optical remote-sensing images is investigated.The proposed method is evaluated on the STGAN and SEN12MS-CR Winter datasets.On the STGAN dataset,the signal-to-noise ratio and structural similarity are 26.706 and 0.759,respectively,which are 9.855 and 0.171 higher than those yielded by the comparison method on average.On the SEN12MS-CR Winter dataset,the signal-to-noise ratio and structural similarity are 28.779 and 0.798,respectively,which are 7.683 and 0.124 higher than those yielded by the comparison method on average.Experimental results show that the proposed method is superior to the comparison method and can remove clouds in optical remote-sensing images more effectively(Tables 1 and 2).The cloud-removed image yielded by this method offers three advantages:1)high color fidelity;2)favorable textural-detail preservation;3)considerable ability to remove shadows caused by clouds(Figs.6 and 7).Second,we discuss the effects of the cross-covariance attention mechanism and gated linear units on a noiseestimation network.Experiments show that the cross-covariance attention not only improves the noise estimation ability of the noise-estimation network but also significantly reduces the computational complexity of the model.The gated linear unit effectively reduces the computational complexity of the network and enhances the cloud-removal ability of the model(Table 3).Finally,a model using a single loss L_(2) and another model using joint loss(L_(2)+LSSIM)were compared.The model using the joint loss achieved better results.Compared with the single loss L_(2),the joint loss enhances the global-structurerecovery ability of the model and improves the quality of the cloud-free image generated by the model(Table 4).Conclusions Cloud removal in remote-sensing images is mandatory in the preprocessing of remote-sensing images and has been investigated extensively.This paper proposes a method to remove clouds from optical remote-sensing images using a hidden diffusion model and restore their surface information.In the forward process,a structural-similarity constraint loss is introduced to alleviate the randomness of model generation,and a U-Net-like cross-covariance attention noise estimation network(NEUTViT)is proposed to estimate the noise distribution more accurately.The cloud removal results obtained on two datasets outperform that of other single-image remote sensing cloud removal methods.The LDMCR model proposed herein performs better than other similar methods;however,it presents some limitations.For example,it can not easily reconstruct surface information enshrouded by large,thick clouds and does not use additional data as aid.In the future,we will use auxiliary data(such as SAR images)and combine them with the cloud-removal tasks of large-scale optical remote-sensing images to investigate cloud removal from optical remote-sensing images using the latent diffusion model.