基于Spark的分布式大数据并行化聚类方法研究

Parallel Clustering Method of Distributed Big Data Based on Spark

为提高异构复合存储数据库中分布式大数据的检索识别能力,提出一种基于Spark融合聚类的异构复合存储数据库中的分布式大数据并行化聚类方法,构建异构复合存储数据库存储结构模型,提取数据库中的分布式大数据的统计序列特征量,采用相空间结构重组方法进行分布式大数据的信息特征挖掘,在高维相空间中实现分布式大数据关联信息检测和特征挖掘。对提取的特征量采用Spark融合聚类方法实现并行聚类,结合自适应的学习算法实现数据聚类中心的自动搜索,提高聚类的收敛性。仿真结果表明,采用该方法进行异构复合存储数据库中分布式大数据聚类的自适应性能较好,聚类准确度较高,误分率较低,具有很好的数据库检索和特征识别能力。

In order to improve the ability of retrieval and identification of distributed big data in heterogeneous composite storage database,a parallel clustering method of distributed big data in heterogeneous compound storage database based on Spark fusion clustering is proposed.The storage structure model of heterogeneous composite storage database is constructed,the statistical sequence features of distributed big data in the database are extracted,and the information feature mining of distributed big data is carried out by using the method of phase spatial structure reengineering.Distributed big data association information detection and feature mining are implemented in high-dimensional phase space.Spark fusion clustering method is used to realize parallel clustering,and adaptive learning algorithm is used to realize the automatic search of data clustering center,which improves the convergence of clustering.The simulation results show that the proposed method has better adaptive performance,higher clustering accuracy,lower misclassification rate and better database retrieval and feature recognition ability for distributed big data clustering in heterogeneous composite storage database.