一种基于YARN的高优先级作业调度实现方案

A Kind of High-priority Job Scheduler Implementation based on YARN

提交到YARN上的一个大数据作业会被切分为一个或者多个任务,任务是大数据作业申请资源和执行的基本单位[1]。在某些领域中存在需要对作业紧急度作有效区分使得高紧急度作业优先获得资源的需求,但是在现有的YARN资源调度策略中,对于提交到YARN上的高优先级作业缺乏资源优先分配和高质量的资源保障机制。本文在修改YARN原有资源调度方案的基础上,提出了一种基于YARN的高优先级作业调度实现方案。实验表明,提交到YARN上的高优先级作业执行效率提升了7%左右,证明设计方案行之有效。

A bigdata job submitting on YARN will be cut into one or more tasks,the task is the unit of execution and applying for resource. There is a need to work as effectively distinguish the urgency of job such a high degree of urgency job has priority to scheduling resource, in certain areas, however, the existing YARN resource scheduling policy, for submission to the high-priority jobs lack of allocation resources on priority and high-quality protection mechanism. Based on modify YARN original resource scheduling scheme, we proposed a high-priority jobs YARN implementation. Experiments show the efficiency of the high-priority job promoting about 7 percent, to prove it's a effectively designer.