大数据分析中基于MapReduce的空间权重创建方法研究

Research on construction method of spatial weights based on mapreduce in analysis of big data

大数据空间分析是Cyber-GIS的重要方面。如何利用现有的网络基础设施(比如大规模计算集群)对大数据进行并行分布式空间分析仍然是一大难题。为此,提出一种基于MapReduce的空间权重创建方法。该方法依托Hadoop框架组织计算资源,基于MapReduce模式从大规模空间数据集中高效创建出空间权重:大空间数据被分为多个数据块,将映射器分布给计算集群中的不同节点,以便在数据中寻找出空间对象的相邻对象,由约简器从不同节点处收集相关结果并生成权重文件。利用Amazon公司弹性MapReduce的Hadoop框架,从人工空间数据中创建基于邻近概念的权重矩阵进行仿真。实验结果表明,该方法的性能优于传统方法,解决了大数据的空间权重创建问题。

Spatial analysis of big data is a key component of Cyber-GIS. However, how to exploit existing cyber infrastructure( e. g. large computing clusters) in performing parallel and distributed spatial analysis on Big data remains a hugechallenge. To solve this problem, a construction method of spatial weights based on MapReduce is proposed in this paper,which creates spatial weights from veiy large spatial datasets efficiently by using computing resources that are organized inthe Hadoop framework: the big spatial data is firstly chunked into pieces, then the mappers are distributed to differentnodes in the computing cluster to find neighbors of spatial objects in the data, and finally the reducers collect the resultsfrom different nodes to generate the weights file. To test the performance of this algorithm, we design experiment to createcontiguity-based weights matrix from artificial spatial data using Amazon, s Hadoop framework called Elastic MapReduce.The experimental results show that the performance of the proposed method is better than the traditional method, and solvethe construct problem of spatial weight in big data.