基于Spark的极限学习机算法并行化研究

Study of ELM Algorithm Parallelization Based on Spark

极限学习机算法虽然训练速度较快,但包含了大量矩阵运算,因此其在面对大数据量时,处理效率依然缓慢。在充分研究Spark分布式数据集并行计算机制的基础上,设计了核心环节矩阵乘法的并行计算方案,并对基于Spark的极限学习机并行化算法进行了设计与实现。为方便性能比较,同时实现了基于Hadoop MapReduce的极限学习机并行化算法。实验结果表明,基于Spark的极限学习机并行化算法相比于Hadoop MapReduce版本的运行时间明显缩短,而且若处理数据量越大,Spark在效率方面的优势就越明显。

Extreme learning mechine(ELM)has high training speed,but with lots of matrix operations,it rernams poor efficiency while applied to massive amount of data.After thorough research on parallel computation of Spark resilient distributed dataset(RDD),we proposed and implemented a parallelized algorithm of ELM based on Spark.And for convenienceof performance comparison,Hadoop-MapReduce-based version was also implemented.Experimental results show that the training efficiency of the Spark-based ELM parallelization algorithm is significantly improved than the Hadoop-MapReduce-based version.If the amount of data processed is greater,the advantage of Spark in efficiency is more obvious.