基于节点集计算能力差异的Hadoop自适应任务调度算法

Hadoop adaptive task scheduling algorithm based on computation capacity difference between node sets

针对异构集群任务推测式执行算法存在的任务进度比例固定、落后任务被动选取等问题,提出基于快慢节点集计算能力差异的自适应任务调度算法。该算法量化节点集计算能力差异实现分集调度,并通过节点与任务速率的动态反馈及时更新快慢节点集,提高节点集资源利用率与任务并行度。在两节点集中,利用动态调整任务进度比例判别落后任务,主动选择采用替代执行方式为落后任务执行备份任务的快节点,从而提升任务执行效率。与最长近似结束时间(LATE)算法的实验对比结果表明,该算法在短作业集、混合型作业集、出现节点性能下降的混合型作业集执行时间上比LATE算法分别缩短了5.21%、20.51%、23.86%,启用的备份任务数比LATE算法明显减少。所提算法可使任务主动适应节点差异,在减少备份任务的同时有效提高作业整体执行效率。

Aiming at the problems of the fixed task progress proportions and passive selection of slow tasks in the task speculation execution algorithm for heterogeneous cluster,an adaptive task scheduling algorithm based on the computation capacity difference between node sets was proposed. The computation capacity difference between node sets was quantified to schedule tasks by fast and slow node sets,and dynamic feedback of nodes and tasks speed were calculated to update slow node sets timely to improve the resource utilization rate and task parallelism. Within two node sets,task progress proportions were adjusted dynamically to improve the accuracy of slow tasks identification,and the fast node which executed backup tasks dynamically for slow tasks by substitute execution implementation was selected to improve the task execution efficiency. The experimental results showed that,compared with the Longest Approximate Time to End（ LATE） algorithm,the proposed algorithm reduced the running time by 5. 21%,20. 51% and 23. 86% respectively in short job set,mixed-type job set and mixed-type job set with node performance degradation,and reduced the number of initiated backup tasks significantly. The proposed algorithm can make the task adapt to the node difference,and improves the overall job execution efficiency effectively with reducing slow backup tasks.