面向混合负载的集群资源弹性调度

Cluster elastic resource scheduling for mixed workloads

针对目前集群资源调度方法难以适应互联网业务多样化、定制化特征的问题,提出了一种面向混合负载的集群资源弹性调度方法。该方法通过构建作业约束描述语言,允许作业基于自身负载特征提出多维度的资源申请和具有负载意识的资源调度算法,实现在同一集群内各类业务统一部署与管理,及时匹配资源需求的变化;通过建立作业的软约束与硬约束之间的转化机制,满足作业在不同执行阶段对资源的定制化需求。实验表明,该方法相比于Hadoop,可允许作业利用较少资源获得更优性能,在实际生产系统中,基于该方法可将集群资源利用率由62%提升到75%。

Aiming at the current situation that existing cluster resource scheduling methods cannot meet the diversified and customized requirements of the Internet services,an approach for elastic cluster resource scheduling for mixed workloads was studied.Firstly,a dynamic description language of job constraints was constructed to describe the multi-dimensional resource requirements from various jobs with different workloads.Secondly,a workload-aware resource scheduling algorithm was used to deploy and make overall plan for various business applications in a single cluster,which could respond to the changes in resource requirements in time.Thirdly,a convertible mechanism for soft and hard constraints was designed for meeting customized resource requirements from different running stages of applications.The experimental results show that the above-mentioned method can outperform the Hadoop in terms of consuming less resources and achieving better performance when running jobs.In real production systems,it can increase the resource utilization from 62％ up to 75％.