基于MapReduce数据密集型负载调度策略研究

Research on Scheduling Strategy of Data Intensive Workloads Based on MapReduce

针对云计算环境中大规模数据集的处理,MapReduce集群已成为一个强大的处理平台。文中提出了一种基于虚拟化平台动态资源重配置的资源评价和动态资源重新配置调度算法。该算法动态地评估作业在截止时间内完成所需要的Map和Reduce计算资源数量,并在不违反用户设定的时间目标的情况下,通过动态地增加或减少独立虚拟机的方式来调整CPU资源,以实现提高数据本地性,同时提高系统在运行作业时的资源利用率。仿真实验结果表明,该算法可以使集群上的MapReduce作业的吞吐率有明显的提高。

Aiming at processing of large-scale data set in cloud computing environment, MapReduce has become a powerful processing platform. In this paper,propose a resource evaluation and dynamic resource reconfiguration and scheduling algorithm based on virtualization platform dynamic resource reconfiguration. It can dynamically evaluate the required number of Map/Reduce slots for every job to meet completion time guarantee and adjust the CPU resources while not violating completion time goals of the users by dynamically increasing or decreasing individual VMs to maximize data locality and also to maximize the use of resources within the system among the active jobs. Simulation results show that the algorithm can improve the throughput of MapReduce jobs on the cluster significantly.