Spark平台加权分层子空间随机森林算法研究

Research of Random Forest Algorithm Using Weighted Stratified Subspace Based on Spark Platform

如何在各式大数据中更快更准确地挖掘有用信息是研究热点。随机森林算法作为一种重要的机器学习算法,适用于大部分数据集。随机森林算法可以并行运行,这是随机森林算法处理大数据集时的优势。将随机森林算法应用在大数据处理框架Spark上,提高了随机森林算法处理大数据集时的速度。首先对随机森林进行参数调优,找到当前数据集的最优参数组合,采用随机森林模型对特征进行重要度计算,筛选掉噪声数据;然后采用卡方检验对数据集的特征进行分层,实现分层子空间随机森林并验证准确率和袋外精度;最后在传统分层子空间随机森林基础上对分层子空间进行加权改进。实验证明改进后的随机森林算法准确率提高了3%,袋外估计精度提高了1%。

How to find useful information out of all kinds of big data faster and more accurately becomes an import problem in the time.As an important machine learning algorithm,random forest algorithm is flexible and suitable for most data sets.The random forest algorithm can run in parallel,this is an advantage when dealing with large data sets.The application of random forest algorithm to big data processing framework Spark can greatly improve the speed of running and processing big data of random forest algorithm.Firstly,the parameter of the random forest were optimized to find the optimal combination of parameters of the current data set.The importance of features are calculated to delete the useless feature by random forest model.Then,chi-square test is used to stratify the features of the data set to achieve the verification accuracy and out-of-bag accuracy of random forest using stratified subspace.Finally,on the basis of the traditional random forest using stratified subspace,the stratified subspace is improved by weighting.The experimental results show that the improved random forest algorithm improves the prediction accuracy by 3% and the out-of-bag estimation accuracy by 1%.