一种基于Spark的改进随机森林算法

AN IMPROVED RANDOM FOREST ALGORITHM BASED ON SPARK

提出一种改进随机森林算法(SP-RF)。通过建立数据抽样索引表和随机特征索引表来实现随机森林算法在Spark上的并行化;通过计算随机森林算法中每个决策树的AUC值来给分类能力不同的决策树分配权重;提高随机森林算法在投票环节的分类精度。实验结果表明改进后的随机森林算法分类精度平均提高5%,运行时间平均减少25%以上。

This paper proposes an improved random forest algorithm(SP-RF).Parallelization of random forest algorithms on Spark was realized by establishing data sampling index table and random feature index tables;it distributed weights to decision trees with different classification abilities by calculating the AUC value of each decision tree in the random forest algorithm;it also improved the classification accuracy of random forest algorithm in the voting process.The experimental results show that the improved random forest algorithm has an average accuracy of 5%higher in classification and an average reduction of more than 25%in running time.