摘要
为了解决在面对海量数据时机器学习算法很难在有效时间内完成规定的任务,并且很难有效地处理高维度、海量数据等问题,提出了基于Hadoop分布式平台的谱聚类算法并行化研究。利用MapReduce编程模式,将传统的谱聚类算法进行重新编写;在该平台上用Canopy算法对数据进行预处理,以达到更好的聚类效果。实验结果表明了设计的分布式聚类算法在加速比等方面有良好的性能,并且在数据伸缩率方面效果明显,改进后的算法适合处理海量数据。
To solve the issues of mass data that algorithms of machine learning can not complete the required tasks within the validity period and are difficult to processing high latitudes and mass data effectively, a parallelization research of spectral clustering algorithm based on Hadoop distributed platform is proposed. The traditional spectral clustering algorithm is re-writed by using the MapReduce programming model, while a Canopy algorithm is used for data preprocessing on the platform to achieve better clustering results. Experimental results verify that distributed clustering algorithm had good performance in speedup and some other aspects, had high ratio of elongation and the improved algorithm is much suitable for processing mass data.
出处
《计算机工程与设计》
CSCD
北大核心
2014年第5期1683-1687,共5页
Computer Engineering and Design
基金
国家自然科学基金项目(61163025)
内蒙古自然科学基金项目(2012MS0912)
内蒙古教育厅高校科研基金项目(njzy12110)
关键词
海量数据
机器学习
聚类算法
谱聚类
分布式框架
massive data
machine learning
clustering algorithm
spectral clustering~ distributed ~ramework