基于Hadoop的PM_(2.5)遥感监测系统设计与实现

Design and Implementation of PM_(2.5) Remote Sensing Monitoring System Based on Hadoop

遥感定量反演的对象为多源遥感数据,处理过程不可避免地涉及到海量数据处理、信息提取和分析.遥感反演既是计算密集型,同时又是数据密集型的科学应用.随着反演理论和技术的发展,地表遥感模型复杂化,数据量呈爆炸式增长,相应地对计算能力的要求也越来越高.同时,随着我国空间技术的不断发展,卫星数据正以指数级的形式迅速增长,对相关应用系统的存储和计算能力也提出了很高的要求.基于以上原因,采用云计算框架,利用Hadoop平台设计该PM_(2.5)卫星遥感监测系统.其中,第一部分介绍了整个系统的架构设计,自上而下包括四个部分;第二部分对该系统的核心算法进行了详细的阐述,利用HDFS和MapReduce分别实现了对海量数据的冗余存储和并行化处理;第三部分进行了性能分析及实例测试,通过详细的对比,可以发现MapReduce并行计算在很大程度上能够提高系统的运行效率;最后对本系统进行总结和展望.通过对"高分一号"卫星数据获得的PM_(2.5)产品进行加速比分析,验证了系统具有高处理效率和高可用性等优点.

The objects of remote sensing quantitative retrieval are multi-source remote sensing data, and the process inevitably involves massive data processing, information extraction and analysis. Quantitative retrieval from remotely sensed data is a data intensive scientific application, where the complexities of processing and analyzing large volumes of data sets have drastically increased computation and data demands. At the same time, with the development of space technology of China, remote sensing data are growing exponentially, which have put forward a high request of the storage and computation capabilities of related application systems too. Based on the above reasons, this paper designed the PM2.5 remote sensing monitoring system by using the framework of cloud computation and Hadoop platform. In this paper, the first part introduces the architectural design of the overall system, four layers from the top layer to hottom; the second part descrihes the core algorithm of this system in detail, and by using HDFS and MapReduce technology, the redundant storage and parallel processing of massive data are achieved; the third part shows performance analysis and examples test, and through the detailed comparison, we can find the efficiency of the system is greatly improved by using MapReduce parallel processing computation; the last part presents summary and prospect of the system. Through the speedup analysis of PM2.5 product from GF-1 Satellite, it validates the high processing efficiency and availability of our system.