并行SVM算法在Flink平台的应用研究

Application Research of Parallel SVM Algorithm on Flink Platform

在大数据时代背景下,数据规模成指数级增长,传统支持向量机(SVM)已无法适应大数据环境,所以需要将传统支持向量机算法改进使其可以应用于大数据计算框架.针对计算过程中存在占用内存大、寻优速度慢等问题,提出一种基于Flink平台的并行支持向量机算法.该方法首先基于层叠支持向量机(Cascade SVM)的合并策略以及训练结构,通过Flink分布式计算框架实现;其次,通过优化并行操作算子的性能引入分布式广播变量,优化算法,有效解决单机SVM算法训练效率低的问题.实验结果表明,结合Flink框架实现SVM算法并行化,能有效的减少了训练时间,提高模型的训练效率.

In the context of the era of big data,the scale of data has grown exponentially,and traditional support vector machines(SVM)have been unable to adapt to the environment of big data,so the traditional support vector machine algorithm needs to be improved so that it can be applied to the big data computing framework.Aiming at the problems of large memory occupation and slow optimization speed in the calculation process,a parallel support vector machine algorithm based on Flink platform is proposed.This method is based on the cascading support vector machine(Cascade SVM)merge strategy and training structure,and is implemented through the Flink distributed computing framework.Secondly,the distributed broadcast variables are introduced by optimizing the performance of the parallel operation operator,and the algorithm is optimized to effectively solve the problem of low training efficiency of the stand-alone SVM algorithm.Experimental results show that the parallelization of SVM algorithm combined with Flink framework can effectively reduce the training time and improve the training efficiency of the model.