大数据Hadoop框架核心技术对比与实现

Comparison and Implementation of Core Technologies of Big Data Hadoop Framework

大数据技术平台有很多,最为流行的是Hadoop生态系统,而Hadoop生态系统涉及的技术繁杂,最为核心是HDFS和MapReduce,前者实现的是分布式存储,后者实现的分布式并行处理,而后又出现了Spark、Hive、Pig等技术,给很多初学者理解和掌握大数据平台造成了困难。为了帮助区分这些技术的用途、优势以及之间的关联性,选择合适技术来解决大数据的计算和存储问题。以经典的WordCount程序为例,在伪分布式集群上使用不同的技术来实现该程序,且易于被还原,通过对实验流程、核心代码、适用场景进行分析研究,总结每种技术的优势和特点,理顺大数据关键技术之间的关系。

Among many big data technology platforms,the most popular one is Hadoop ecosystem.The technology of Hadoop ecosystem is complex,and the core is HDFS and MapReduce,the former realizes distributed storage,while the latter realizes distributed parallel processing.Spark,Hive,Pig and other technologies appear in the ecosystem,which makes difficult for many beginners to understand and master them.In order to distinguish the use,advantages and relevance of these technologies and choose appropriate technologies to solve the problem of big data,this paper uses the classic wordcount program on the pseudo distributed cluster.It can be easily restored by using different technologies.By analyzing the experimental process,core code and applicable scenarios,the advantages and characteristics of each technology are summarized,and the relationship between key technologies of big data is straightened out.