Angel^+:基于Angel的分布式机器学习平台

Angel^+:A Large-Scale Machine Learning Platform on Angel

【目的】随着大數据时代的来临,数据变得高维、稀疏,机器学习模型也变得复杂、高维,因此也给分布式机器学习系统带来了很多挑战。尽管研究人员已经开发了很多高性能的机器学习系统,比如TensorFlow、PyTorch、XGBoost等,但是这些系统存在以下两个问题:(1)不能与现有的大数据系统很好的结合;(2)不够通用,这些系统往往是为了某一类机器学习算法设计。【方法】为了解决以上两个挑战,本文介绍Angel^+:—个基于参数服务器架构的分布式机器学习平台。【结果】Angel^+能够高效的支持现有的大数据系统以及机器学习系统——浪赖于参数服务器处理高维模型的能力,Angel^+能够以无侵入的方式为大数据系统(比如Apache Spark)提供高效训练超大机器学习模型的能力,并且高效的运行已有的分布式机器学习系统(比如PyTorch)。此外,针对分布式机器学习中通信开销大和掉队者问题,Angel^+也提供了模型平均、梯度压缩和异构感知的随机梯度下降解法等。【结论】笔者结合Angel^+开发了很多高效、易用的机器学习模型,并且通过实验验证了Angel^+平台的高效性。

[Objective]Real-world data becomes much more complex,sparse and high-dimensional for the big data shock in this era.According to this,modem ML models are designed in a deep,complicated way,which arises challenges when designing a distributed machine learning(ML)system.Though researchers have developed many efficient centralized ML systems like TensorFlow,PyTorch and XGBoost,these systems suffer from the following two problems:(1)They cannot integrate well with existing big data systems,(2)they are not general enough and are usually designed for specific ML models.[Methods]To tackle these challenges,we introduce Angel^+,a large-scale ML platform based on parameter servers.[Results]With the power of parameter servers,Angel+can efficiently support existing big data systems and ML systems without neither breaking the core of big data systems,Apache Spark for instance,nor degrades the computation efficiency of current ML frameworks like PyTorch.Furthermore,Angel^+ provides algorithms like model averaging,gradient compression and heterogeneous-aware stochastic gradient descent,to deal with the huge communication cost and the straggler problem in distributed training process.[Conclusions]We also enhance the usability of Angel^+ by providing efficient implementation for many ML models.We conduct extensive experiments to demonstrate the superiority of Angel^+.