一种分布式架构的空间大数据服务平台

A Spatial Big Data Service Platform Based on Distributed Architecture

随着位置信息在各行各业中的广泛应用,空间大数据得到迅猛发展.空间大数据除具有数据量大的特点之外,还具有复杂性,同时,越来越多的应用对数据的实时性也有较高的要求.传统的GIS软件在承载和处理空间数据时,也面临越来越多的挑战,如难以对复杂多样的空间数据进行一体化存储和管理;传统GIS软件架构和单机处理能力,无法对较大体量(10亿条记录或更大)的空间数据进行分析.本文从分布式存储技术、分布式空间处理计算技术和分布式计算协调技术三个方面阐述如何应对上述问题,并提出了将Spark分布式框架和Su-perMap iObject for Spark空间处理引擎相结合的分布式空间处理计算技术,以及数据库的一体化管理和监控技术,实现对多种数据库如PostgreSQL集群、MongoDB和Elasticsearch的统一管理和监控.

With the widespread use of location information in all aspects of life, spatial big data has been developed rapidly. In addi-tion to the characteristics of large data amounts, spatial big data is also complex, and needs to meet the real-time data requirements of various applications. Traditional GIS software are facing more and more challenges in carrying and processing spatial big data. For example, it's difficult to integrate and manage complex and diverse spatial big data, and the traditional GIS software architecture and stand-alone processing ability is unable to be used for analyzing large volume (1 billion records or more) of spatial big data. This pa-per discusses how to deal with the above problems from three aspects:distributed storage technology, distributed spatial processing and computing technology, and distributed computing coordination technology. This paper proposes a distributed spatial processing and computing technology combining Spark and SuperMap iObjects for Spark, and an integrated management and monitoring technology of the database to realize the unity of multiple databases including PostgreSQL cluster, MongoDB and Elasticsearch.