摘要
为提高小作业的执行效率及系统资源利用率,分析MapReduce各个阶段的执行过程,提出一种基于任务时间的任务调度算法。针对现有的Reduce任务分配算法的不足,综合考虑Map任务执行的剩余时间及Reduce任务shuffle和sort阶段所需时间两个因素,对等待队列中的作业进行重排序,依次调度队列中作业的Reduce任务。实验结果表明,该算法有效提高了小作业的执行效率及系统的资源利用率。
To improve the efficiency of small jobs and system resource utilization,the processes of various stages of MapReduce were analyzed,and a task scheduling algorithm based on task time was proposed.Considering the remaining time of Map task as well as the time of shuffle phase and sort phase in Reduce task,jobs were resorted in the waiting queue,and Reduce task was assigned in order.Experimental results show that the algorithm can effectively improve small jobs＇ efficiency and the resource utilization efficiency.
出处
《计算机工程与设计》
北大核心
2016年第3期675-678,756,共5页
Computer Engineering and Design
基金
国家自然科学基金项目(61233003)
关键词
分布式计算
小作业
任务分配
剩余时间
资源利用率
distributed computing
small job
task allocation
remaining time
resource utilization
