基于Hadoop平台的作业调度算法研究

Study on the job scheduling algorithm based on Hadoop Platform

爆炸式增长的数据要求高效率地使用计算资源,Hadoop是解决大数据处理的一个方向,不过Hadoop平台还处于发展中,很多地方还有待完善。对Hadoop自带的三种调度器的优缺点进行分析,针对希望区分和动态改变作业服务质量的情况提出了动态优先调度算法。实验结果表明动态优先调度算法能够在作业的运行中动态改变作业的优先级,在效率上也有所提升。

The explosive growth of data requires the efficient use of computing resources e. Hadoop, though very important for mass data processing, still needs to be improved. The advantages and disadvantages of the three sched- ulers of Hadoop are analyzed and, in view of the attempt to distinguish and dynamically improve the job service quali- ty, dynamic priority scheduling algorithm is proposed. The experimental results show that dynamic priority scheduling algorithm can dynamically change the priority of the job in its operation and improves its efficiency.