Hadoop云平台下的聚类算法研究

Research on clustering algorithm on Hadoop platform

为了解决在面对海量数据时机器学习算法很难在有效时间内完成规定的任务,并且很难有效地处理高维度、海量数据等问题,提出了基于Hadoop分布式平台的谱聚类算法并行化研究。利用MapReduce编程模式,将传统的谱聚类算法进行重新编写;在该平台上用Canopy算法对数据进行预处理,以达到更好的聚类效果。实验结果表明了设计的分布式聚类算法在加速比等方面有良好的性能,并且在数据伸缩率方面效果明显,改进后的算法适合处理海量数据。

To solve the issues of mass data that algorithms of machine learning can not complete the required tasks within the validity period and are difficult to processing high latitudes and mass data effectively, a parallelization research of spectral clustering algorithm based on Hadoop distributed platform is proposed. The traditional spectral clustering algorithm is re-writed by using the MapReduce programming model, while a Canopy algorithm is used for data preprocessing on the platform to achieve better clustering results. Experimental results verify that distributed clustering algorithm had good performance in speedup and some other aspects, had high ratio of elongation and the improved algorithm is much suitable for processing mass data.