基于动态分布式聚类算法的大数据查询处理方法

Big Data Query Processing Method Based on Dynamic Distributed Clustering Algorithm

针对现有大数据空间查询处理方法存在执行时间长和查询结果不够准确的问题,提出一种基于动态分布式聚类算法的大数据查询处理方法,该方法分为数据预处理、数据聚类和查询处理3个部分.首先将输入数据划分为多个子集,以RRD格式存储在一组机器节点中;其次采用划分和层次混合动态聚类算法,在Apache Spark平台上对数据进行分布式聚类;最后通过K近邻查询方式获得高精度和高效率查询结果.实验结果表明,本文提出的方法具有可扩展性,可为空间查询处理提供高质量的结果,比其他查询方法更具优势.

Aiming at the problems of long execution time and inaccurate query results in existing big data spatial query processing methods,a big data query processing method based on dynamic distributed clustering algorithm has been proposed,which includes data pre-processing,data clustering and query processing.Firstly,the method divides the input data into multiple subsets and stores them in a group of machine nodes in RRD format.Secondly,the partition and hierarchical hybrid dynamic clustering algorithm is used to cluster the data on Apache spark platform.And lastly,the high-precision and high-efficiency query results are obtained by K-Nearest Neighbor query.The experimental results show that the proposed method is scalable,and provides high quality results for spatial query processing,which has more advantages than other query methods.