一种基于改进累积方差百分比的红外高光谱数据降噪方法

Research on the Noise Reduction with Hyper-Resolution Infrared Spectrum Based on Improved PCV Method

降低红外高光谱观测数据中的噪声水平是提高温湿廓线反演精度和反演稳定性的重要环节。采用主成分分析法降噪时,最优主成分个数k的选择一般是根据统计和经验的方法确定。统计的方法大都是根据累积方差百分比法,通过人为设定累积贡献率阈值确定最优主成分个数,使得该方法具有较大的主观性和随意性;经验的方法则需要实时的等效噪声光谱（NESR）数据做标准化处理将非均匀噪声转化为高斯分布,而实时的NESR数据在很多情况下不易获取。针对上述问题,提出了一种基于改进累积方差百分比的主成分降噪方法,通过迭代计算选取不同主成分时重构光谱辐射与模拟光谱辐射的偏差来计算累积贡献率阈值,根据阈值确定最优的主成分个数。该方法解决了确定累积贡献率阈值主观随意性的问题,并且不需要实时的NESR数据做标准化处理。根据物理反演结果分析了数据的标准化对降噪的影响,结果表明,标准化对降噪效果的影响很小,由标准化造成的k值计算误差对降噪效果的影响更大。利用该方法对2011年4个季度中具有代表性的数据做降噪处理,反演的温度廓线均方根误差相比于经验公式法在0.32~3km高度上提高了约0.1K,与利用等效噪声光谱标准化后的降噪数据的反演结果精度相当。在无法获取等效噪声光谱数据情况下,该方法可以客观合理地对地基红外高光谱数据进行降噪。

The noise reduction with observed high resolution infrared radiance is crucial to improve the accuracy and stability of the retrieval of thermodynamic profiles.When applying the principal component analysis noise filter algorithm to the observed radiance,the optimal number k of principal components that used in the algorithm was mostly calculated with the statistical and empirical method.The percent cumulative variance method is one of the statistical methods that have been commonly used to calculate k,however,the threshold of the percent cumulative variance was determined subjectively and arbitrarily,which limits the application of this method.While the empirical method need the real-time Noise-Equivalent Spectral Radiance（NESR）to normalize non uniform noise in the observed data,but the real-time NESR needs the raw data of complex spectrum which is not easy to obtain in most cases.Aiming at the solving the problems above,a PCA noise filter based on the Improved PCV algorithm is proposed,of which the threshold is determined by iteratively calculating the difference between the simulated and reconstructed spectrum using different principal components,whereby kis determined such that the PCV is larger than the threshold.The new method solves the problem of arbitrary of the determination of k,and at the same time it doesn＇t need the real-time NESR to normalize the observed radiance.First,the impact of normalization on the noise reduction is analyzed using physical retrieval of temperature profiles;the result shows that the impact is very small,which less than the impact of calculation error of kis caused by normalization on the retrieval of temperature profiles.Then,the noise reduction of the representative radiance data which covers four quarters of 2011 shows that,the RMSE of the retrieved temperature profile using the Improved PCV method is improved by 0.1Kcompared to the factor indicator function method when the real-time NESR is not available,and it is almost the same with the latter when the normalization is done.Under the condition that the NESR is not available,the method proposed in this article could objectively and reasonably reduce the noise level of the ground-based high resolution infrared radiance.