基于Mesos的分布式参数优化调度策略及系统设计

Distributed Parameter Optimization Scheduling Strategy and System Design Based on Mesos

本文针对分布式参数优化系统的调度策略进行研究,并实现了一个基于Mesos的分布式参数优化系统。利用Mesos的资源接口,把多种常见的参数优化算法和任务调度封装为一个可以在Mesos上运行的框架软件。并针对Mesos的两级调度机制,提出了一种对混合部署集群上多作业竞争环境下的分布式参数优化系统的调度优化策略。本文设计了多组实验,对框架软件的资源调度策略与FIFO调度策略进行对比测试,可以满足多租户在混合部署场景下的使用。降低了在集群环境下进行深度学习等常见场景下的分布式参数优化的难度,在多任务竞争时提高资源使用效率。

This paper implements a distributed parameter optimization system based on Mesos and studies the scheduling strategy of this system. Using the resource interface of Mesos, the system packages a variety of common parameter optimization algorithms and task scheduling strategy into a framework software that can run on Mesos. Aiming at the two-level scheduling mechanism of Mesos, a dynamic scheduling strategy for distributed parameter optimization system in multi-job environment on hybrid deployment cluster isproposed. This paper designs several experiments, and compares the resource scheduling strategy of the architecture software with the FIFO scheduling strategy in the hybrid deployment scenario. This work reduces the difficulty of optimizing distributed parameters in common scenarios such as deep learning in a cluster environment, and improves resource utilization efficiency in multi-task environment.