基于云计算的数据挖掘系统设计

Design of Data Mining System Based on Cloud Computing

为了高效、快速地解决呈指数增长的数据处理问题,提高数据储存、运算能力,文中提出了基于云计算的数据挖掘系统的设计。该系统首先分析了主流云计算平台Spark的组件构成和运行机制,深入研究其计算架构的编程原理。同时利用Spark进行了C4.5算法和K-medoids聚类算法的并行化设计,有效提高算法的运行速度、收敛速度和结果的稳定性。测试表明,在进行海量数据的分析处理时,文中提出的云计算平台在分类误差内,可有效提高整体系统的运算速度,分类效率也大幅提高。

In order to solve exponentially increasing data processing problems and improve data storage and computing power efficiently and quickly, this paper proposed a cloud computing-based data mining system design. The system first analyzed the component composition and operation mechanism of the mainstream cloud computing platform Spark, and deeply studied the programming principle of its computing architecture. At the same time, Spark was used to parallelize the C4.5 algorithm and K-medoids clustering algorithm, which effectively improved the running speed, convergence speed and stability of the algorithm. The test showed that in the analysis and processing of massive data, the cloud computing platform proposed in this paper could effectively improve the computing speed of the whole system and improve the classification efficiency.