期刊文献+
任意字段
题名或关键词
题名
关键词
文摘
作者
第一作者
机构
刊名
分类号
参考文献
作者简介
基金资助
栏目信息

一种多层次Hadoop平台设计

A Multi-Layer Hadoop Platform
下载PDF
导出
摘要 大数据和大数据中心规模的进一步扩大为Hadoop平台的可扩展性提出了新的要求.在系统地分析了制约Hadoop平台性能和可扩展性因素的基础上,提出了一种多层次Hadoop平台.通过将Hadoop平台划分为若干区域,形成全局-区域-节点的多层次结构.由Master节点负责系统级的元数据管理和任务分发,区域管理节点负责区域级的元数据管理和任务分发,从而提高Hadoop平台可扩展性和容错性.实验表明,该平台可有效提高Hadoop的可扩展性,具有一定的可行性. With the further expansion of Big Data and Big Data Centre, new extensibility requirements are proposed on the Hadoop Platform. In the base of systematically analysis constraints on performance and extensibility of Hadoop Platform, this paper presents a multi-level Hadoop Platform which has a structure of Global-Region-Node after being divided. In this platform, Master Node is responsible for metadata management and task development on the global level, and regional management node takes charge of metadata management and task development on the regional level, which improve both scalability and fault tolerance of Hadoop Platform. Experiments show that this platform can effectively promote the scalability of Hadoop and has certain feasibility.
作者 李兆兴 马自堂
机构地区 解放军信息工程大学密码工程学院
出处 《微电子学与计算机》 CSCD 北大核心 2016年第2期27-30,共4页 Microelectronics & Computer
关键词 HADOOP NameNode SeMNode 多层次 Hadoop NameNode SeMNode multi-layer
分类号 TP391 [自动化与计算机技术—计算机应用技术]
  • 相关文献

参考文献7

  • 1Tom White T. Hadoop: the definitive guide[M]. O" Reilly: Yahoo Press, 2011 : 35-36.
  • 2Shvachko K, Kuang H, Radia S, et al. The hadoop distributed file system[C]// 2010. Mass Storage Sys- tems and Technologies (MSST), 2010 IEEE 26th Symposium on. Nevada, Indine Village, IEEE, 2010: 1-10.
  • 3覃雄派,王会举,杜小勇,王珊.大数据分析——RDBMS与MapReduce的竞争与共生[J].软件学报,2012,23(1):32-45. 被引量：386
  • 4邓鹏,李枚毅,何诚.Namenode单点故障解决方案研究[J].计算机工程,2012,38(21):40-44. 被引量：27
  • 5Borthakur D, Sarma J. Apache hadoop goes realtime at facebook[C]// Proceedings of the 2011 ACM SIG- MOD International Conference on Management of da- ta. New York,ACM,2011: 1071.
  • 6Tom White. The small files problem[EB/OL]. (2009- 02-02). [ 2015-04-04 ]. http..//www, cloudear, corn/ blog/2009/02/02/the-small-files-problem/.
  • 7王凯.MapReduce集群多用户作业调度方法的研究与实现[D].国防科学技术大学,2010.11.

二级参考文献84

  • 1Zhou MQ, Zhang R, Zeng DD, Qian WN, Zhou AY. Join optimization in the MapReduce environment for column-wise data store. In: Fang YF, Huang ZX, eds. Proc. of the SKG. Ningbo: IEEE Computer Society, 2010.97-104. [doi: 10.1109/SKG.2010.18].
  • 2Afrati FN, Ullman JD. Optimizing joins in a Map-Reduce environment. In: Manolescu I, Spaecapietra S, Teubner J, Kitsuregawa M, Leger A, Naumann F, Ailamaki A, Ozcan F, eds. Proc. of the EDBT. Lausanne: ACM Press, 2010. 99-110. [doi: 10.1145/ 1739041.1739056].
  • 3Sandholm T, Lai K. MapReduce optimization using regulated dynamic prioritization. In: Douceur JR, Greenberg AG, Bonald T, Nieh J, eds. Proc. of the SIGMETRICS. Seattle: ACM Press, 2009. 299-310. [doi: 10.1145/1555349.1555384].
  • 4Hoefler T, Lumsdaine A, Dongarra J. Towards; efficient MapReduce using MPI. In: Oster P, ed. Proc. of the EuroPVM/MPI. Berlin: Springer-Verlag, 2009. 240-249. [doi: 10.100'7/978-3-642-03770-2_30].
  • 5Nykiel T, Potamias M, Mishra C, Kollios G, Koudas N. MRShare: Sharing across multiple queries in MapReduce. PVLDB, 2010, 3(1-2):494-505.
  • 6Kambatla K, Rapolu N, Jagannathan S, Grama A. Asynchronous algorithms in MapReduce. In: Moreira JE, Matsuoka S, Pakin S, Cortes T, eds. Proc. of the CLUSTER. Crete: IEEE Press, 2010. 245-254. [doi: 10.1109/CLUSTER.2010.30].
  • 7Polo J, Carrera D, Becerra Y, Torres J, Ayguad6 E, Steinder M, Whalley I. Performance-Driven task co-scheduling for MapReduce environments. In: Tonouchi T, Kim MS, eds. Proc. of the 1EEE Network Operations and Management Symp. (NOMS). Osaka: IEEE Press, 2010. 373-380. [doi: 10.1109/NOMS.2010.5488494].
  • 8Zaharia M, Konwinski A, Joseph AD, Katz R, Stoica I. Improving MapReduce performance in heterogeneous environments. In: Draves R, van Renesse R, eds. Proc. of the ODSI. Berkeley: USENIX Association, 2008.29-42.
  • 9Xie J, Yin S, Ruan XJ, Ding ZY, Tian Y, Majors J, Manzanares A, Qin X. Improving MapReduce performance through data placement in heterogeneous Hadoop clusters. In: Taufer M, Rfinger G, Du ZH, eds. Proc. of the Workshop on Heterogeneity in Computing (IPDPS 2010). Atlanta: IEEE Press, 2010. 1-9. [doi: 10.1109/IPDPSW.2010.5470880].
  • 10Polo J, Carrera D, Becerra Y, Beltran V, Torres J, Ayguad6 E. Performance management of accelerated MapReduce workloads in heterogeneous clusters. In: Qin F, Barolli L, Cho SY, eds. Proc. of the ICPP. San Diego: IEEE Press, 2010. 653-662. [doi: 10.1109/ ICPP.2010.73].

共引文献418

微电子学与计算机
2016年 第2期

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部