新型分布式计算系统中的异构任务调度框架

Heterogeneous Task Scheduling Framework in Emerging Distributed Computing Systems

随着大数据和机器学习的火热发展,面向机器学习的分布式大数据计算引擎随之兴起.这些系统既可以支持批量的分布式学习,也可以支持流式的增量学习和验证,具有低延迟、高性能的特点.然而,当前的一些主流系统采用了随机的任务调度策略,忽略了节点的性能差异,因此容易导致负载不均和性能下降.同时,对于某些任务,如果资源要求不满足,则会导致调度失败.针对这些问题,提出了一种异构任务调度框架,能够保证任务的高效执行和被执行.具体来讲,该框架针对任务调度模块,围绕节点的异构计算资源,提出了概率随机的调度策略resource-Pick_kx和确定的平滑加权轮询算法.Resource-Pick_kx算法根据节点性能计算概率,进行概率随机调度,性能高的节点概率越大,任务调度到此节点的可能性就越高.平滑加权轮询算法在初始时根据节点性能设置权重,调度过程中平滑加权,使任务调度到当下性能最高的节点上.此外,对于资源不满足要求的任务场景,提出了基于容器的纵向扩容机制,自定义任务资源,创建节点加入集群,重新完成任务的调度.通过实验在benchmark和公开数据集上测试了框架的性能,相比于原有策略,该框架性能提升了10%-20%.

With the rapid development of big data and machine learning,the distributed big data computing engine for machine learning have emerged.These systems can support both batch distributed learning and incremental learning and verification,with low latency and high performance.However,some of them adopt a random task scheduling strategy,ignoring the performance differences of nodes,which easily lead to uneven load and performance degradation.At the same time,for some tasks,if the resource requirements are not met,the scheduling will fail.In response to these problems,a heterogeneous task scheduling framework is proposed,which can ensure the efficient execution and execution of tasks.Specifically,for the task scheduling module,the proposed framework proposes a probabilistic random scheduling strategy resource-Pick_kx and a definite smooth weighted round-robin algorithm around the heterogeneous computing resources of nodes.The resource-Pick_kx al-gorithm calculates the probability according to the performance of the node,and performs random scheduling with probability.The higher the probability of a node with high performance,the higher the possibility of task scheduling to this node.The smooth weighted round-robin algorithm sets the weights according to the node performance at the beginning,and smoothly weights during the scheduling process,so that the task is scheduled to the node with the highest performance.In addition,for task scenarios where resources do not meet the requirements,a container-based vertical expansion mechanism is proposed to customize task resources,create nodes to join the cluster,and complete task scheduling again.The performance of the framework is tested on benchmarks and public data sets through ex-periments.Compared with the current strategy,the performance of the proposed frame is improved by 10% to 20%.