期刊文献+

基于参数优化的Hadoop云计算平台 被引量:2

Hadoop Cloud Computing Model Based on Parameters Optimization
下载PDF
导出
摘要 传统的数据分析,很难满足现阶段大数据处理效率的要求.Hadoop云计算技术的应用,实现了海量数据存储和分析,提高了数据存储和分析的效率.在总结传统系统利弊的基础上,以Hadoop分布式文件系统(HDFS)取代现有的单机数据存储,以map/reduce应用程序取代传统的单机数据分析,并对其做出优化.实验证明,Hadoop系统架构在生产上部署、投入使用的可行性. The traditional data analysis, it is difficult to meet data processing efficiency requirements at this stage. Application of Hadoop cloud computing technology, realization of data storage and analysis, improve the efficiency of data storage and analysis. This paper is on the basis of summing up the pros and cons of traditional system, with Hadoop Distributed File System (HDFS) to replace the existing stand-alone data storage, map/reduce application instead of the traditional stand-alone data analysis, and make the optimization. The experiment approves that the feasibility of Hadoop system architecture in the production deployment and using.
作者 李寒 唐兴兴
出处 《计算机系统应用》 2013年第3期21-24,共4页 Computer Systems & Applications
关键词 云计算 HADOOP 数据分析 MAP REDUCE cloud computing Hadoop data analysis map/reduce
  • 相关文献

参考文献8

  • 1TOM Wbite. Hadoop: The Definitive Guide. US: O'Reilly. 2005.
  • 2Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters. Communications of the ACM, 2005,51(1): 107-113.
  • 3Dhruba B. The Hadoop Distributed File System: Architecture and Design.2007.
  • 4Dean J, Ghemawat S. Distributed programming with Mapreduce. In: Oram A, Wilson G, eds. Beautiful Code. Sebastopol: O'Reilly Media, Inc., 2007: 371-384.
  • 5李丽英,唐卓,李仁发.基于LATE的Hadoop数据局部性改进调度算法[J].计算机科学,2011,38(11):67-70. 被引量:17
  • 6丁光华,周继鹏,周敏.基于MapReduce的并行贝叶斯分类算法的设计与实现[J].微计算机信息,2010,26(9):190-191. 被引量:5
  • 7李应安.基于MapReduce的聚类算法的并行化研究.微计算机信息,2010,9.
  • 8Hadoop. http://wiki.apache.org/hadoop/Hbase/PerformanceEv aluation.

二级参考文献22

  • 1张冬慧,孙波,徐照财,程显毅.文本自动分类关键技术研究[J].微计算机信息,2008,24(6):197-199. 被引量:12
  • 2Dean J, Ghemawat S.MapReduce: Simplifed Data Processing on Large Clusters[C]//Proc. of the 6th Symposium on Operating System Design and hnplementation, San Francisco. 2004.
  • 3Christopher D. Manning, Prabhakar Raghavan and Hinrich Schutze. Introduction to Information Retrieval. Cambridge University Press. 2008.
  • 4Cutting D. Scalable Computing with MapReduce [C]//Proc. of O'Reilly Open Source Convention, Poland. 2005.
  • 5Tom M.Mitchell.曾华军,张银奎等译.机器学习[M].北京:机械工业出版社.2003.
  • 6Cheng-Tao Chu, Sang Kyun Kim, Yi-An Lin. Map-Reduce for Machine Learning on Multicore. [C]//Proceedings of Neural Information Processing Systems Conference (NIPS). Vancouver, Canada. 2006.
  • 7David Lewis. Na i ve(bayes) at forty:The independence assumption in information retrieval. [C]//In ECML98: Tenth European Conference On Machine Learning. Chemitz, Germany. 1998.
  • 8Vaquero L M,Rodero-Merino I.,Caceres J, el al. A break in the cloud: Towards a Cloud Definition [J].ACM SIGCOMM Computer Communication Review, 2009,39 ( 1 ) : 50-55.
  • 9Vaquero L M,Rodero-Merino L,Caceres J, et al. A break in the cloud: Towards a Ckoud Definition[J].ACM SIGCOMM Computer Communication Review, 2009,39 ( 1 ) : 50-45.
  • 10Crovella M, HarchobBalter M, Murta C D. Tasassignment in a distributed system:Improving performance by unbalancing load [M]. Measurement and Modeling of Computer Systems, 1998: 268-269.

共引文献20

同被引文献17

  • 1Tom White.Hadoop权威指南[M].2版.北京:清华大学出版社,2011.
  • 2Boutaba R,Cheng L,Zhang Q.On cloud computational models and the heterogeneity challenge[J].Journal of Internet Services and Applications,2012,3(1):77.
  • 3Bass L,Kazman R,Ozkaya I.Open Source Systems:Grounding Research[M].Berlin:Springer,2011:50-61.
  • 4[美]Lam C.Hadoop实战[M].韩冀中,译.北京:人民邮电出版社,2011.
  • 5Shkapenyuk V,Suel T.Design and implementation of a high-performance distributed web crawler[M].San Jose:IEEE,2002.
  • 6Boldi P,Codenotti B,Santini M,et al.Ubicrawler:a scalable fully distributed web crawler[J].Software:Practice and Experience,2004,34(8):711-726.
  • 7吴百锋,彭澄廉,赵立勇.并行和分布式计算机监测系统的实现原理[J].计算机学报,2010,20(3):23-27.
  • 8董超群,司马超,吴利,等.云计算:概念,现状及关键技术[C]∥全国高性能计算学术年会论文集.无锡:中国计算机学会,2008:15-18.
  • 9陶冶,刘建勋,唐明董.基于Map/Reduce的分布式Web服务搜索引擎设计与实现[J].计算机科学,2011,38(8):183-192.
  • 10许笑,张伟哲,张宏莉,方滨兴.广域网分布式Web爬虫[J].软件学报,2010,21(5):1067-1082. 被引量:25

引证文献2

二级引证文献13

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部