期刊文献+
任意字段
题名或关键词
题名
关键词
文摘
作者
第一作者
机构
刊名
分类号
参考文献
作者简介
基金资助
栏目信息

一种Hadoop中基于改进遗传算法的作业调度算法 被引量：4

A Job Scheduling Algorithm Based on Improved Genetic Algorithm in Hadoop
下载PDF
导出
摘要 Hadoop平台是一个分布式系统基础框架,用户可以在不了解底层细节的情况下,开发分布式程序,并且可以充分利用集群来高速运算和存储。Hadoop平台核心技术之一作业调度技术,主要职责就是控制作业执行的顺序以及计算资源的分配,这直接影响到Hadoop平台的整体性能和资源利用情况。文中基于遗传算法框架对作业调度问题进行了研究,提出了一种新的分段编码策略,在编码时将计算资源作为遗传操作的基本单元,以此为基础提出新的区域杂交算子和变异算子。实验证明,此算法是Hadoop平台下有效的作业调度算法。 Hadoop platform is a distributed system framework, users who don＇ t need to know the underlying details of the case can devel op distributed programming and take full advantage of the cluster to highspeed computing and storing. Job scheduling, as one of the core technologies in Hadoop,its main function is to control the order of job execution and the allocation of computing resources, which directly affect the overall performance and resource utilization of Hadoop. It is to study job scheduling problem based on genetic algorithm, propo sing a new segmentation coding strategies, taking computing resources as the basic unit of the genetic operation. In addition a new regional hybrid operator and mutation operator are included. Experiments show that it is an effective job scheduling algorithm in Hadoop.
作者 徐肖 胡吉明
机构地区 河海大学计算机与信息学院
出处 《计算机技术与发展》 2013年第3期10-13,18,共5页 Computer Technology and Development
基金 国家社科基金项目(08BSH031)
关键词 HADOOP平台 作业调度 遗传算法 Hadoop job scheduling genetic algorithm
分类号 TP311 [自动化与计算机技术—计算机软件与理论]
  • 相关文献

参考文献11

  • 1丁辉,张大华,罗志明.基于Hadoop的海量数据处理平台研究[C]//2011电力通信管理暨智能电网通信技术论坛论文集.出版地不祥:出版者不详,2011.
  • 2Dean J, Ghemawat S. MapReduce: Simplified Data Processing on Large Clusters [ J ]. Communications of the ACM, 2008,51 (1) :107-113.
  • 3夏袜.Hadoop平台下的作业调度算法研究与改进[D].广州:华南理工大学,2010.
  • 4李玲娟,张敏.云计算环境下关联规则挖掘算法的研究[J].计算机技术与发展,2011,21(2):43-46. 被引量：48
  • 5李远方,邓世昆,闻玉彪,韩月阳.Hadoop-MapReduce下的PageRank矩阵分块算法[J].计算机技术与发展,2011,21(8):6-9. 被引量：13
  • 6Holland J H. Adaptation in Natural and Artificial System[ M].Ann Arbor, MI : University of Michigan Press, 1975.
  • 7Jin C, Vecchiola C, Buyya R. Mrpga : An extension of mapre- duce for parallelizing genetic algorithms [ C ]//IEEE Fourth International Conference on eScience. [ s. 1.] : [ s. n. ] ,2008: 214-221.
  • 8Maruyama T, Hirose T, Konagaya A. A fine- grained parallel genetic algorithm for distributed parallel systems [ C ]//Pro- ceedings of the 5th International Conference on Genetic Algo- rithms. San Francisco, CA, USA : Morgan Kanfmann Publishers Inc. , 1993 : 184-190.
  • 9Sastry K, Goldberg D E, Llora X. Towards billion-bit optimiza- tion via a parallel estimation of distribution algorithm [ C ]// GECCO' 07 : Proceedings of the 9th Annual Conference on Ge- netic and Evolutionary Computation. New York, NY, USA: ACM ,2007:577-584.
  • 10王小平 曹立明.遗传算法[M].西安：西安交通大学出版社,2002..

二级参考文献24

  • 1刘华元,袁琴琴,王保保.并行数据挖掘算法综述[J].电子科技,2006,19(1):65-68. 被引量：15
  • 2Weiss A. Computing in Clouds[ J]. ACM Networker,2007,11 (4) : 18-25.
  • 3Buyya R, Yeo C S, Venugopal S. Market-Oriented Cloud Computing : Vision, Hype, and Reality for Delivering IT Services as Computing Utilities[ C ]//Proceedings of the 2008 10^th IEEE International Conference on High Performance Computing and Communications. [ s. l. ] : [ s. n. ] ,2008 : 5-13.
  • 4Apache. Hadoop [ EB/OL]. 2006. http://lucene, apache. org/hadoop/.
  • 5Dean J, Ghemawat S. Mapreduce: Simplified data processing on large clusters [ C ]//Proceedings of the 6th Symposium on Operating System Design and Implementation. San Francisco, California, USA : USENIX Association, 2004 : 137-150.
  • 6Wu X, Kumar V, Ghosh R J, et al. Top 10 algorithms in data mining[J]. Knowledge and Information Systems,2008,14 (1) :1-37.
  • 7Agrawal R, Sharer J C. Parallel Mining of Association Rules [ J]. IEEE Transactions on Knowledge and Data Engineering, 1996,8 ( 6 ) : 962- 969.
  • 8Aflori C, Craus M. Grid implementation of the Aprioti algorithm[ J]. Engineering Software,2007, 38( 5): 295-300.
  • 9Dean J, Ghemawat S. MapReduce: Simplied Data Proessing on Large Clusters[ C] JJProceedings oi the 6th Conference on Symposium on Operating Systems Design & Implementation. [ s. 1. ] : USENIX Association, 2004.
  • 10Catanzaro B C, Sundaram N, Keutzer K. A Map Reduce Framework for Programming Graphics Processors [ C ]//Work- shop on Software Tools for MultiCore. [s. l. ]: Is. n. ] ,2006.

共引文献183

同被引文献35

  • 1ApacheHadoop. Hadoop [ EB/OL ]. 2015 - 12-28. http ://ha- doop. apache, org.
  • 2He C, Lu Y, Swanson D. Matchmaking: a new MapReduce scheduling technique [ C ]//Proceedings of IEEE third interna- tional conference on cloud computing technology and science. [ s. 1. ] :IEEE,2011:40-47.
  • 3Zaharia M, Borthakur D, Sarma J S, et at. Job scheduling for multi- user mapreduce clusters [R]. Berkeley: University of California ,2009.
  • 4Raj A, Kaur K, Dutta U, et at. Enhancement of Hadoop clus- ters with virtualization using the capacity scheduler [C]// Third international conference on services in emerging mar- kets.[s. 1. ] :IEEE,2012:50-57.
  • 5Zhang Xiaohong, Feng Yuhong, Feng Shengzhong, et al. An ef- fective data locality aware task scheduling method for MapRe-duce framework in heterogeneous environments [ C ]//Proceed- ings of international conference on cloud and service compu- ting. Hong Kong: IEEE ,2011:235-242.
  • 6Ibrahim S, Jin H, Lu L, et al. Maestro : replica- aware map scheduling for mapreduce [ C ]//Proceedings of 12th IEEE/ ACM international symposium on cluster ,cloud and grid com- puting. Ottawa, Canada : IEEE ,2012:435-442.
  • 7Zaharia M, Borthakur D, Sen Sarma J, et al. Delay scheduling : a simple technique for achieving locality and fairness in cluster scheduling[ C ]//Proceedings of the 5th European conference on computer systems. New York, NY, USA : ACM, 2010 : 265 - 278.
  • 8Sandholm T, Lai K. Dynamic proportional share scheduling inHadoop [ C ]//Job scheduling strategies for parallel process- ing. Berlin : Springer,2010 : 110-131.
  • 9Kurazumi S, Tsumura T, Saito S, et al. Dynamic processing slots scheduling for I/O intensive jobs of Hadoop MapReduce [C]//Proceedings of the third international conference on networking and computing. [ s. 1. ] : [ s. n. ] ,2012:288-292.
  • 10Kc K,Anyanwu K. Scheduling Hadoop jobs to meet deadlines [ C]//Proceedings of the 2nd IEEE international conference on cloud computing technology and science. [ s. 1. ] : IEEE, 2012:388-392.

引证文献4

二级引证文献4

计算机技术与发展
2013年 第3期

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部