资源不均衡Spark环境任务调度优化算法研究

Task scheduling optimization in Spark environment with unbalanced resources

由于硬件资源的更新换代,集群中各个节点的计算能力会变得不一致。集群异构的出现导致集群计算资源不均衡。目前Spark大数据平台在任务调度时未考虑集群的异构性以及节点资源的利用情况,影响了系统性能的发挥。构建了集群节点的评价指标体系,提出利用节点的优先级来表示其计算能力。提出的节点优先级调整算法能够根据任务执行过程中节点的状态动态调整各个节点的优先级。基于节点优先级的Spark动态自适应调度算法(SDASA)则根据实时的节点优先级值完成任务的分配。实验表明,SDASA能够缩短任务在集群中的执行时间,从而提升集群整体计算性能。

Due to the updating of hardware resources,the computing capacity of nodes in a cluster becomes inconsistent.The emergence of cluster heterogeneity leads to the imbalance of cluster computing resources.At present,Spark big data platforms does not consider the cluster heterogeneity and the node resource utilization in task scheduling,which affects the system performance.This paper constructs a performance evaluation index system of cluster nodes,and proposes to use the priority of nodes to express their computing capacity.The proposed node priority adjustment algorithm can dynamically adjust the priority of each node according to the status of the node during task execution.The Spark Dynamic Adaptive Scheduling Algorithm(SDASA)based on node priority completes task assignment according to the real-time node priority.Experiments show that SDASA can shorten the execution time of tasks in the cluster and improve the overall computing performance of the cluster.