基于MapReduce的K_means并行算法及改进

Parallel K-Means Algorithm and Improved Based on Map Reduce

针对传统k_means聚类算法在处理海量数据时所面临的内存不足、运算速度慢等问题,提出了一种基于Map Reduce的K_means并行算法,同时为了改善k_means算法在初始值确定方面的盲目性,采用canopy算法进行改进.实验结果表明,基于Map Reduce的K_means并行算法和改进后的算法均能产生良好的聚类效果,不仅提高了聚类质量,而且在处理大数据集方面,改进后的算法的还能够得到趋近于线性的加速比.

In view of the problems that traditional k-means clustering algorithm faces in dealing with mass data, such as running out of memory, the operating in slow speed and so on, this paper proposes a parallel k-means algorithm based on MapReduce. At the same time, in order to overcome the blindness of the k-means algorithm in terms of determining the initial value, we use the canopy algorithm to improve the insufficient. The experimental results show that the parallel k-means algorithm based on MapReduce has an effect on clustering before and after the improvement, not only the quality of the clustering has been increased, but in terms of processing large datasets. The speed-up ratio of the improved algorithm can get closer to the linear.