应用程序属性感知的Yarn资源调度模型研究

Research on application attribute aware Yarn resource scheduling model

Hadoop应用程序存在计算密集属性和调度时间属性,但是Hadoop大数据平台集成的第二代资源管理器Yarn内置的三种资源调度器无法将相同属性的应用程序均衡分配到计算节点上,导致部分节点负载过高,出现严重的计算任务长尾效应。文中提出了一种应用程序属性感知的Yarn负载均衡调度模型——APB Scheduler。APB Scheduler自动感知应用程序属性,将相同属性应用程序的Container按照动态资源计划均衡分配到集群计算节点上,并使用NSGA-Ⅲ算法完成最优分配方案计算。实验结果表明,APB Scheduler解决了相同属性应用程序的Container分配倾斜问题,大幅提升了集群的性能和稳定性。

Hadoop applications have computation intensive and scheduling time attributes.However,the three built-in resource schedulers of the second generation resource manager,which is integrated in Hadoop big data platform,are unable to evenly distribute applications with the same attributes to the computing nodes,resulting in excessive load on some nodes and serious long tail effect of computing tasks.This paper presents an application attribute aware yarn load balancing scheduling model-APB Scheduler.APB Scheduler automatically perceive the application attributes,evenly allocate the containers of the same attribute application to the cluster computing nodes according to the dynamic resource plan,and use the NSGA-III algorithm to complete the calculation of the optimal allocation scheme.Through experimental verification,APB Scheduler solves the container allocation skew problem of applications with the same content,and greatly improves the performance and stability of the cluster.