遥感大数据分布式技术研究与实现

Research and Implementation of Remote Sensing Big Data Distributed Technology

面向卫星遥感海量数据,针对其数据量的急速增长,对数据分析、价值挖掘提出了全新的挑战,引入驱动大数据应用的分布式模式,建立了适应卫星遥感大数据的网格模型,打破了数据的时空割裂和限制,数据可以作为整体进行存储、计算和应用,模型设计的网格、时间片、物理层的基本结构,可以保证未来云计算的实施。该文提出了基于希尔伯特曲线的网格散列算法,以此建立的分布式系统具有优异的并行读写性能和良好的负载均衡能力;遥感大数据分布式系统,实现了数据的高速分布式并行读写,支持数据的精确时空匹配和动态获取,整个系统的扩展能力可以达到线性增长,系统基于通用软硬件平台实施,实现卫星遥感大数据灵活、按需和简便的应用。

In the past ten years, the global various digital information grows explosively, and a big data era with massive data production, sharing and application is opened. In this decade, with the development of information technology, distributed storage and computing technology get great development to deal with the explosive growth of information, and the knowledge system and technical reserves are established gradually. In China, research on big data and distributed computing is being carried out widely. For satellite remote sensing data of large volume and rapid growth, the traditional archive-callback-application cannot meet demands of data analysis and data mining in the era of big data. The traditional file-based way has many limitations, especially when used for cloud computing and intelligent services, and it is very difficult to use. The big data grid model and distributed model is the key to solve the bottleneck, enabling real-time computing and on-demand services, and therefore it has important reference significance. It overcomes the temporal and spatial fragmentation problem, making the remote sensing data possible to be stored, calculated and applied as a whole. Based on the Hilbert curve grid hash algorithm, a distributedsystem containing fundamental structure of grid, time slice and physical layer is established, demonstrating excellent parallel read-write performance. Hilbert hash algorithm has stable discrete degree, which is the key for the grid model to maintain spatial correlation and to map two-dimensional space to one-dimensional sequence. Using the distributed system, instead of traditional way of data file organization and management, properties flexible and intuitive data acquisition are realized. Users can truly experience a new way of what you see is what you get and what you get is what you need to get. The future system which is based on the data model, will greatly increase the work efficiency, make the focus from the data itself to data applications. Internet-based cloud computing grid cell calculation can be realized, and the extension ability of the whole system can achieve linear growth, based on the general hardware and software platform. The implementation of this system will greatly improve the work efficiency, completing high-speed parallel data reading and writing, making on-demand data application more smoothly.