Hadoop平台下基于截止时间限制的动态调度算法的研究

Dynamic scheduling algorithm based on deadline for Hadoop platform

为了满足有截止时间限制的MapReduce作业的需求,提出一种基于截止时间限制的动态调度算法(DCDS)。该算法实时监控作业运行状况,并对作业运行时间进行动态估算,从而确定作业优先级;对于时间紧迫的作业,可通过抢占策略来保证在用户要求的截止时间内完成。实验结果表明,与Hadoop平台现有的调度算法相比,该算法不仅能满足作业截止时间的要求,也提高了系统资源的利用率和吞吐量。

In order to satisfy the deadline for MapReduce jobs, this paper presented a dynamic scheduling algorithm based on deadline （DCDS）. This algorithm could monitor the real-time implementation procedure of jobs, and estimated the time of jobs dynamically to compute the priority of the job. For emergency jobs, this algorithm used preemptive strategy to ensure jobs com- pleted under the deadline. Experimental results show that the scheduling algorithm not only meets the deadline of the job, but also improves the efficiency and throughput of the system resources compared with the existing Hadoop scheduling algorithm.