基于负载均衡的MapReduce后备任务上限自适应算法

Upper limit value of backup task self-adaptive algorithm of MapReduce based on load balance

已有算法采用固定后备任务上限,不能动态适应负载水平变化。针对该问题,提出了基于负载均衡的MapReduce后备任务上限自适应算法。通过计算空闲节点强度和网络带宽分析系统负载水平,不断调整后备任务上限,精确控制后备任务数量,避免因过多空闲节点空载导致资源浪费或过度执行后备任务导致网络拥塞。实验表明,该算法能有效感知系统负载水平,对后备任务数量作出合理调整,并且比原算法在负载均衡和作业响应时间上有明显的提升。

Current algorithms fixed the upper limit value of backup task, it can not dynamically adapt to change in load level. To solve this problem, this paper proposed a upper limit value of backup task self-adaptive algorithms based on the load bal- ance. The algorithm constantly changed the backup task numbers to control the number of backup tasks precisely by calculating idle TaskTracker intensity and network bandwidth, this could avoid the resources waste which caused by much idle TaskTrack- er or the congestion which caused by the excessive execution of backup task. Experimental results show that the algorithm can aware system load levels effectively, adjust the number of backup tasks legitimately, and the load balance and response time of the job has improved significantly than the original algorithm.