基于Spark的分布式青藏高原MODIS LST插值方法实现研究

A Spark-based Distributed Implementation of Interpolating MODIS LST over Qinghai-Tibetan Plateau

受云等诸多因素的影响,青藏高原的MODIS LST数据通常存在大面积数据缺失,传统的插值方法很难达到理想效果,因此学者们研究了许多新方法,其中较好的一种是利用和缺失像元具有相似的LST变化特征的已知像元集估算缺失像元的LST,能够能实现高精度插值,但计算花费巨大,通常需要借助高性能计算机完成。将分布式计算引擎Spark应用于该方法,用一个普通计算机群来代替高性能计算机实现对青藏高原MODIS LST数据快速有效的插值,并对比分析了不同硬件条件、不同数据尺度下原方法和基于Spark实现的该方法两者的性能。结果表明:基于Spark的实现方案有效可行;当节点数和数据量较少时,后者的性能低于前者;随着硬件资源和数据量的增加,后者的性能表现更好并逐渐超过前者;使用新版的Spark编写代码或着将插值方法的代码编译成.so动态库再通过Spark调用,可以进一步提高该方案的插值性能。

The MODIS LST products are often obscured by clouds and other atmospheric disturbances,resulting in severe data loss.Traditional interpolation methods cannot be effectively applied when there is large area of missing data.Many methods are developed to solve this problem.A better approach is to estimate the LST of missing pixels by using a known set of pixels with a similar LST variation feature as the missing pixel.But it usually has to be done with supercomputers.In order to make the above method free from supercomputers computer,a distributed implementation of the method is proposed by combing it and the Spark that is a distributed computing engine.The interpolation efficiency of two model is compared under different hardware resources and data.The results show that the scheme is effective and feasible,the performance of the proposed method is lower than the Yu method when with a small amount of data and the cluster nodes,but with the increase of hardware resources,the performance of the proposed method is better than that of the Yu method.In addition,the performance of the scheme can be further improved by using the newest Spark or compiling the code of the Yu method into a .so library and then using the Spark call it.