基于Spark的扩展孤立森林算法并行化改造实验设计

Experimental design of Spark-based parallelization transformation of extended isolated forest algorithm

针对海量或高维数据进行异常检测实验时,往往检测速度较慢、效率较低。针对此问题,设计了一种基于Spark分布式计算的扩展孤立森林异常检测算法改造实验。实验基于Spark框架,分别在数据抽样、训练、预测等阶段设计并行化改造方法,通过与单核条件下的算法对比,验证了并行化方法在保证准确性的前提下执行效率得到大大提高。此实验对加深学生对大数据分布式并行处理知识的理解,引导其对海量数据挖掘相关技术的学习兴趣具有积极作用。

When performing anomaly detection experiments on massive or high-dimensional data,the detection speed is often slow and the efficiency is low.For this problem,a comprehensive experimental design of distributed machine learning for parallelizing the extended isolated forest anomaly detection algorithm is proposed,relying on the high-performance advantage of Spark distributed computing.Based on the Spark framework,the experiment designs parallel transformation methods in data sampling,training,prediction stages.Compared with the algorithm running on a single core,it is verified that the execution efficiency of parallelization method is greatly improved under the premise of ensuring accuracy.This experiment plays a positive role in deepening students'understanding of distributed parallel processing knowledge of big data and guiding their interest in learning massive data mining related technologies.