基于Hadoop和MapReduce的大数据处理系统设计与实现

Designing and realizing a big data processing system based on Hadoop and MapReduce

随着信息技术的猛烈发展,各行各业产生的数据呈指数型增长。如何利用,分析,整合这些"大数据"成为信息时代的一大挑战。文章针对信息时代下,大规模数据处理分析效率等一系列相关问题,利用Hadoop技术,设计并实现了一套基于Hadoop和MapReduce的大数据处理系统,该系统利用Hadoop分布式文件系统(HDFS)存储数据,利用Hadoop Map Reduce框架分布式处理数据分析任务。实验结果表明:该系统可以将单一数据处理任务分配到一个Hadoop集群上去,继而提高大数据文件处理效率。

With the development of information technology,data generated from all walks of life are growing exponentially.The utilization,analysis and integration of the big data has become a challenging issue to the information era.In order to address the above issues,this paper utilizes Hadoop technique and presents a big data processing system based on Hadoop and MapReduce.The proposed system utilizes Hadoop MapReduce to distribute the data processing and stores data by taking advantage of the HDFS features.The experimental results show that our system is able to enhance the efficiency of the big data processing by distributing a single data processing procedure to a Hadoop cluster.