基于奖惩共存收益模式的大数据作业调度器

BIG data job scheduler based on reward and punishment coexistence profit model

为满足服务商获得最大收益、达到平台资源利用率最大的要求,提出一种基于奖惩共存收益模式的大数据作业调度器,该调度器中包括基于任务执行时间的确定轮数算法(TRN)和基于最大轮数的作业调度算法(MRNS)。TRN确定作业在不同奖惩阶段的Map和Reduce的最大轮数组合以及最大标准时间;MRNS选择具有局部最大收益的作业和该作业的任务最大轮数方案,制定出基于任务的作业调度策略。实验结果表明,提出的作业调度器对比已有的调度器,作业平均完成时间缩短了13.5%~25.9%、服务商收益提高了16.3%~26.4%,平台资源利用率平均提高了7.8%~10.3%,故该大数据作业调度器具有一定的高效性和可用性。

In the big data computing environment, an efficient big data job scheduler based on reward and punishment coexistence profit model was developed that enabled service providers to achieve maximum profit and empowered the resource of the platform to reach optimum utilization. The job scheduler consisted of the determining algorithm of round number based on task execution time(TRN) and the job scheduling algorithm based on maximum round number(MRNS). The TRN algorithm determined maximum standard time and maximum round number combination plans of Map tasks and Reduce tasks in the different rewards or penalties stage. The MRNS algorithm sorted the jobs with the local maximum profits and the maximum round number plans of tasks. Experimental results show that the scheduler is more efficient and available than the existed scheduler in terms of ave-rage completion time of jobs that shortened by 13.5%-25.9%, the profit of service providers that increased by 16.3%-26.4% and platform average resource utilization that increased by 7.8%-10.3%.