期刊文献+

FlowS:一种MapReduce数据流公平调度方法 被引量:4

FlowS:A Fair Scheduling Method for Mapreduce Dataflow
下载PDF
导出
摘要 MapReduce Job的调度机制一直是学术研究的热点。在分析MapReduce数据流调度模型的基础上,提出一种面向MapReduce数据流的公平调度方法FlowS。该方法采用数据流池来分配资源以保证MapReduce数据流的隔离性,并且采用数据流池动态构建算法来确保资源的公平分配。实验表明,该调度方法可以有效提高Hadoop集群对MapReduce数据流的处理效率。 MapReduce Job scheduling has been paid great attention in academic research.Based on the analysis of MapReduce dataflow scheduling model,this paper presented a fair scheduling method for MapReduce dataflow-FlowS.This method can not only provide the isolation of MapReduce dataflow through dataflow pools,but also assure the fairness of resource allocation through dynamic construction algorithm.The results of experiences show that the proposed method can improve the processing efficiency of Hadoop Clusters.
出处 《计算机科学》 CSCD 北大核心 2012年第9期157-161,174,共6页 Computer Science
基金 国家自然科学基金项目(61170074) 国家科技重大核高基项目(2010ZX01042-001-001-05) 国家科技支撑计划(2011BAH15B05 2012BAH05F02)资助
关键词 MapReduce数据流 资源管理 公平调度方法 MapReduce dataflow Resource management Fair scheduling
  • 相关文献

参考文献18

  • 1Jeffrey D, Sanjay G. MapReduce= Simplified data processing on large clusters[J].Communications of the ACM, 2008, 51 (1): 107-113.
  • 2Welcome to Hadoop MapReduce[EB/OL]. http://hadoop, a- pache, org/mapreduce/.
  • 3Thusoo A,Sarma J S,Jain N,et al. Hive -- A warehousing solu- tion over a MapReduce framework[J]. PVLDB, 2009,2(2) :938- 941.
  • 4Christopher O, Benjamin R, Utkarsh S, et al. Pig Latin: A Not- So-Foreign Language for Data Processing[C]//Proceedings of the 2008 ACM SIGMOD international conference on Manage- ment of Data. New York,USA,g008:1099-1110.
  • 5Cascading[EB/OL]. http://www, cascading, org/.
  • 6Oozie. Hadoop Workflow System [EB/OL]. httpa//issues, a- pache, org/jira/browse/HA-DOOP-5303/.
  • 7Zhang Chen, Hans D S. CloudWF: A computational workflow system for clouds based on Hadoop[C]//Proceedings of the ]st International Conference on Cloud Computing. Germany, 2009: 393-404.
  • 8Zaharia M, Borthakur D, Sarma J S, et al. Job Scheduling for Multi-User MapReduce Clusters[R]. University of California at Berkeley, 2009.
  • 9The hadoop fair scheduler [EB/OL]. http://issues, apache. org/jira/browse/HAIXOOP-ar4fi.
  • 10Capacity Scheduler for Hadoop [EB/OL]. http://Hadoop, a- pache, org/common/docs/current/Capacity _ scheduler, html, 2010-03-22.

同被引文献56

  • 1Zhuo Tang,Junqing Zhou,Kenli Li,Ruixuan Li.A MapReduce task scheduling algorithm for deadline constraints[J]. Cluster Computing . 2013 (4)
  • 2Pengsen Cheng,Junxiu An.The Key as Dictionary Compression Method of Inverted Index Table under the Hbase Database[J]. Journal of Software . 2013 (5)
  • 3Abhishek Verma,Brian Cho,Nicolas Zea,Indranil Gupta,Roy H. Campbell.Breaking the MapReduce stage barrier[J]. Cluster Computing . 2013 (1)
  • 4Yingyi Bu,Bill Howe,Magdalena Balazinska,Michael D. Ernst.The HaLoop approach to large-scale iterative data analysis[J]. The VLDB Journal . 2012 (2)
  • 5Yanfeng Zhang,Qixin Gao,Lixin Gao,Cuirong Wang.iMapReduce: A Distributed Computing Framework for Iterative Computation[J]. Journal of Grid Computing . 2012 (1)
  • 6Zhiwei Xu,Bo Yan,Yongqiang Zou.Beyond Hadoop: Recent Directions in Data Computing for Internet Services[J]. International Journal of Cloud Applications and Computing (IJCAC) . 2011 (1)
  • 7Carstoiu, Dorin,Lepadatu, Elena,Gaspar, Mihai.Hbase - non SQL Database, Performances Evaluation[J]. International Journal of Advancements in Computing Technology . 2010 (5)
  • 8Michael I,Vijayan P,Jon C ,et al. Quincy:Fair scheduling for distributed computing clusters [ C ]//Proceedings of the 22nd ACM SIGOPS Symposium on Operating Systems Principles. US,2009:261 - 276.
  • 9Joel W, Deepak R, Kirsten H, et al. FLEX: A slot allocation scheduling optimizer for MapReduee workloads [ C ]//Proceedings of International Middleware Conference. Germany,2010 : 1 - 20.
  • 10Dodonov E, deMdl R. A novel approach for distributed application scheduling based on prediction of communication events [ J ]. Future Generation Computer Systems,2010,26 (5) :740 - 752.

引证文献4

二级引证文献83

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部