一种面向分布式机器学习的云计算资源调度方法

Resource Scheduling for Distributed Machine Learning in Cloud Computing

在云计算环境下,并发训练多机器学习模型会造成严重的共享集群资源竞争,影响执行效率。针对该问题,论文提出一种面向分布式机器学习的云计算资源调度方法。根据历史监测数据建立迭代次数与模型质量提升间的模型,在线预测资源分配对模型质量提升的影响,制定资源优化调度策略,并且设计了资源调度框架。实验结果表明,所提出的方法能够快速适应任务和负载的动态变化,实现多个模型训练作业的整体性能最大化。

In the cloud computing environment,concurrently training multi-machine learning models can cause serious resource competition of a shared cluster and affect execution efficiency.To address the above problem,this paper proposes a cloud computing resource scheduling method for distributed machine learning.According to the historical monitoring data,the model between the number of iterations and the quality of the model is established.The effect of online resource allocation on the quality improvement of the model is predicted.The resource optimization scheduling strategy and the resource scheduling framework are designed.The experimental results show that the proposed method can quickly adapt to the dynamic changes of tasks and workloads,and maximize the overall performance of multiple models training operations.