期刊文献+

Hadoop平台下计算能力调度算法的改进与实现 被引量:4

Realization and improvement of capacity scheduling algorithm with Hadoop platform
下载PDF
导出
摘要 在计算能力调度算法中没有全面考虑各资源特征的分配是否满足作业多样的服务要求,提出一种基于优先级的计算能力加权调度算法,根据作业的优先级以及提交时间等因素来计算作业的权重。依据作业的权重对作业队列进行排序并分配空闲的slot给队首的作业,从而避免调度陷入局部最优也能更好地满足作业的多样性服务要求。在搭建的Hadoop平台上进行实验表明,改进后的算法能较均衡地分配系统资源减少一些作业的等待时间,并且运行全部作业的用时有所减少。 Among capacity scheduling algorithms, it is not fully taken into account whether the allocation of resources meets job varied service requirements. This paper presents a priority based weighted capacity scheduling algorithm which calculates the weight based on the priority of jobs, job' s wait time and other factors of job. Based on the weight of the job to sort the job queue and assign free slot to the team' s first job, so as to avoid falling into local optimal solution and better meet job varied service requirements. Experimental results on Hadoop platform show that improved algorithm can be more balanced with the resources, reduce waiting time for some jobs, and decrease the total run time of the job.
出处 《计算机工程与应用》 CSCD 北大核心 2015年第19期61-65,共5页 Computer Engineering and Applications
基金 安徽省自然科学基金(No.1308085QF113)
关键词 云计算 作业调度 HADOOP 加权 Map Reduce编程模型 cloud computing job scheduling Hadoop weighting MaPreduce programming model
  • 相关文献

参考文献12

  • 1Dean J,Ghemawat S.Map Reduce:Simplified data processing on large clusters[J].Communications of the ACM,2008,51(1):107-113.
  • 2Ralf L.Google’s Map Reduce programming model-revisited[J].Science of Computer Programming,2008,70(1):1-30.
  • 3Wbite T. Hadoop权威指南[M]. 曾大聃, 周傲英, 译. 北京: 清华大学出版社, 2010.
  • 4Apache Hadoop[EB/OL].[2013-09-01].http://hadoop.apache.org/.
  • 5罗军舟,金嘉晖,宋爱波,东方.云计算:体系架构与关键技术[J].通信学报,2011,32(7):3-21. 被引量:826
  • 6Fair Scheduler[EB/OL].[2013-09-01].http://hadoop.apache.org/common/docs/r0.20.2/fair_scheduler.html.
  • 7Capacity Scheduler[EB/OL].[2013-09-01].http://hadoop.Apache.org/common/docs/r0.20.2/capacity_scheduler.html.
  • 8Torabzadeh E,Zandieh M.Cloud theory-based simulated annealing approach for scheduling in the two-stage assembly flowshop[J].Advances in Engineering Software,2010,41(10):1243-1298.
  • 9Zaharia M,Borthakur D,Sen Sarma J,et al.Delay scheduling:A simple technique for achieving locality and fairness in cluster scheduling[C]//Proceedings of Euro Sys’10.New York,NY,USA:ACM,2010:265-278.
  • 10Matei Z,Dhruba B,Joydeep S,et al.Delay scheduling:A simple technique for achieving locality and fairness in cluster scheduling[C]//Proc of the 5th European Conference on Computer Systems,2010:265-278.

二级参考文献75

  • 1米勒.云计算[M].史美林,姜进磊,孙瑞志,等译.北京:机械工业出版社,2009:125-128.
  • 2FOSTER I, YONG ZHAO, RAICU I, et al. Cloud computing and grid computing 360-degree compared[C] // Proceedings of the 2008 Grid Computing Environments Workshop. Washington, DC: IEEE Computer Society, 2008:1 - 10.
  • 3ARMBRUST M, FOX A, GRIFFITH R, et al. Above the clouds: A Berkeley view of cloud eomputing[EB/OL]. [2010 -01 -25]. http://www, eecs. berkeley, edu/Pubs/TechRpts/20Og/EECS-20og- 28. pdf.
  • 4BARROSO L A, DEAN J, HOLZLE U. Web search for a planet: the google cluster architecture[J]. IEEE Micro, 2003, 23(2) : 22 - 28.
  • 5CHIEN A, CALDER B, ELBERT S, et al. Entropia: Architecture and performance of an enterprise desktop grid system[J]. Journal of Parallel and Distributed Computing, 2003, 63(5):597-610.
  • 6KIM J S, NAM B, MARSH M, et al. Creating a robust desktop grid using peer-to-peer services[EB/OL]. [ 2009 - 10 - 16]. ftp://ftp. cs. umd. edu/pub/hpsl/papers/papers-pdf/ngs07.pdf.
  • 7ABRAHAM A, BUYYA R, NATH B. Nature's heuristics for scheduling jobs on computational grids[ C]// The 8th International Conference on Advanced Computing and Communications. New Delhi: Tata McGraw-Hill Publishing, 2000:45-52.
  • 8DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters[ C]//Proceedings of the 6th Symposium on Operating System Design and Implementation. New York: ACM, 2004:137 - 150.
  • 9The CLOUDS Lab. Gridsim[ EB/OL]. [ 2010 - 06 - 25]. http:// www. cloudbus. org/gridsim/.
  • 10王小平 曹立明.遗传算法[M].西安:西安交通大学出版社,2002..

共引文献1026

同被引文献47

引证文献4

二级引证文献21

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部