期刊文献+
任意字段
题名或关键词
题名
关键词
文摘
作者
第一作者
机构
刊名
分类号
参考文献
作者简介
基金资助
栏目信息

HDFS下载效率的优化 被引量：23

Improving downloading performance in hadoop distributed file system
下载PDF
导出
摘要 针对HDFS的内部数据下载效率较低和可能出现的负载不均衡的问题进行了研究,从分布式文件整体下载效率和数据块的下载效率两方面提出了优化方法。实验结果表明:两个方法都能提高效率,但在集群有大量DataNode的前提下,两者结合起来的方法能更好地提高下载效率和均衡DataNode的负载。 Concerning the problems such as low downloading efficiency and imbalanced load of DataNode in Hadoop Distributed File System HDFS.Inthis paper two methods to optimize were proposed one was to improve the whole process of downloading a file the other was to optimize the downloading a block by a parallel download algorithm for dynamically allocating load by speed.Mathematical analysis and experiments prove that two methods can enhance the efficiency.Meanwhile by combining the two methods downloading is more efficient and more stable when the load of DataNode can be balanced to some extent.
作者 曹宁 吴中海 刘宏志 张齐勋
机构地区 北京大学软件与微电子学院
出处 《计算机应用》 CSCD 北大核心 2010年第8期2060-2065,2240,共7页 journal of Computer Applications
关键词 云计算 Hadoop档案系统(HDFS) 多线程 并行下载 cloud computing Hadoop Distributed File System HDFS multi-thread parallel downloading
分类号 TP301.6 [自动化与计算机技术—计算机系统结构] TP393 [自动化与计算机技术—计算机应用技术]
  • 相关文献

参考文献11

  • 1HADOOP Wi-ki[EB/OL].[2009-07-01].http://wiki.apache.org/hadoop/.
  • 2GHEMAWAT S,GOBIOFF H,LEUNG S T.The google file system.[EB/OL].[2009-07-01].http://labs.google.com/papers/gfs.html.
  • 3DEAN Jean,GHEMAWAT S.Map/reduce:simplified data processing on large clusters[EB/OL].[2009-07-01].http://static.googleusercontent.com/external _ content/untrusted _ dlcp/labs.google.com/zh-CN//papers/mapreduce-osdi04.pdf.
  • 4Map/Reduce[EB/OL].[2009 -07 -01].http://wiki.apache.org/hadoop/HadoopMapReduce.
  • 5HDFS[EB/OL].[2009-07-10].http://wiki.apache.org/hadoop/ProjectDescription.
  • 6Yahoo! Launches world's largest hadoop production application[EB/OL].[2009 -07 -01].http://tinyurl.com/2hgzv7.
  • 7Applications powered by Hadoop[EB/OL].[2009-07-01].http://wiki.apache.org/hadoop/PoweredBy.
  • 8SCHLOSSER S,LIN J.Hadoop Summit 2008[R/OL].[2009-07 -01].http://developer.yahoo.com/events/hadoopsummit2010/agenda.html.
  • 9ZAHARIA M,KONWINSKI A,ANTHONY D.et al.Improving mapreduce performance in heterogeneous environments[C] // 8th USENIX Symposium on Operating Systems Design and Implementation.Washington,DC:IEEE,2008,1:29-42.
  • 10Map/Reduce Tutorial[EB/OL].[2009-07-01].http://hadoop.apache.org/core/docs/r0.20.0/mapred_tutorial.html.

二级参考文献5

  • 1毛德操 胡希明.Linux内核源代码情景分析[M].杭州:浙江大学出版社,2001..
  • 2Robert Love. Linux Kemel Development Second Edition.Sams Publishing, January 2005.
  • 3Daniel P Bovet. Understanding the Linux Kemel. O'Reilly,December 2004.
  • 4Moshe Bar. Linux File Systems. McGraw-Hill Companies,Inc, 2003.
  • 5David A Rusling. The Linux Kemel [N]. http://www.plinux.org/LDP/, 1999.

共引文献2

同被引文献179

  • 1崔杰,李陶深,兰红星.基于Hadoop的海量数据存储平台设计与开发[J].计算机研究与发展,2012,49(S1):12-18. 被引量：141
  • 2许春香,肖国镇.门限多重秘密共享方案[J].电子学报,2004,32(10):1688-1689. 被引量：41
  • 3BRADFORD R, KOTSOVINOS E, FELDMANN A, et al. Live wide - area migration of virtual machines including local persistent state[C C ]//Proc of the 3 rd International Conference on Virtual Execution Environments ,2007 : 169 - 179.
  • 4ALEX C, SNOEREN, HARI BALAKRISHNAN. An end to end approach to host mobility[ C]//Proc of the 6th Annual International Conference on Mobile Com- puting and Networking , 2000 :155 - 166.
  • 5BORTHAKU D. The Hadoop distributed file system: architecture and design [ EB/OL]. (2007 - 07 - 01) [ 2011 - 01 - 17]. http:// hadoop, apache, org/common/docs/r0. 18.2/hdfs_design. pdf.
  • 6GHEMAWAT S, GOBIFF H, LEUNG S-T. The Google file system [C]// Proceedings of 19th ACM Symposium on Operating System Principles. New York: ACM, 2003:29-43.
  • 7DEAN J, GHEMAWAT S. MapReduce: simplifed data processing on large clusters [ C]// OSDI '04: Sixth Symposium on Operating System Design and Implementation. Berkeley: USENIX Association, 2004:137 - 150.
  • 8CHANG F, DEAN J, GHEMAWAT S, et al. Bigtable: a distributed storage system for structured data [ C]//Proeeedings of the 7th USENIX Syrup on Operating Systems Design and Implementation. Berkeley: USENIX Association, 2006:205 - 218.
  • 9ANDROUTSELLIS-THEOTOKIA S, SPINELLIS D. A survey of peer-to-peer content distribution technologies [ J]. ACM Computing Surveys, 2004, 36(4): 335-371.
  • 10POUREBRAHIMI B, BERTELS K, VASS1LIADIS S. A survey of peer-to-peer networks [ C] // Proceedings of the 16th Annum Workshop on Circuits, Systems and Signal Processing. Veldhoven, Netherlands: [s. n. ], 2005:570 -577.

引证文献23

二级引证文献158

计算机应用
2010年 第8期

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部