基于Spark的并行图聚类算法研究

Study of Parallelized Graph Clustering Algorithm Based on Spark

对并行图聚类算法进行了研究。基于Spark提出了一个新的并行图聚类算法;由于Spark中的top操作需要耗费大量的内存,提出了一个新算法来替代top操作,有效减少了所消耗的内存;通过对自底向上的层次聚类算法进行改进提高了聚类的速度;基于图数据的特征提出了一种图数据过滤的方法来减少算法运行的时间以及所占用的空间并对其有效性进行了说明。仿真结果表明,运行效果优于进行比较的其他并行化图聚类算法。

The parallelized graph clustering algorithm is researched.A new parallelized graph clustering algorithm is proposed based on Spark.As the top operation of Spark occupies a lot of memory space,a new algorithm which is used to substitute the top operation is proposed to reduce the memory consumption.By improving bottom up hierarchical clustering algorithm,the speed of the proposed algorithm is improved.A new data filtering method based on the feature of graph data is proposed.By the method,the running time and memory space comsuption is reduced greatly.The reason of the high efficiency of this filtering method is explained.Simulation result indicates that the proposed algorithm is better than other parallelized graph clustering algorithms.