分布式并行编程模型MapReduce及其应用研究

Distributedparallel programming model MapReduce and its application

在传统的并行编程模型中,对大量数据如何进行并行计算、如何为每个任务分发数据、如何处理单点故障等问题,都需要大量的程序分析和设计,这些问题的有效处理都需要程序员显式地使用有关技术来解决.对于程序员来说,这是一项具有极大困难的工作,使得原本简单的运算反而变得非常复杂,这些问题的存在也在一定程度上制约了并行程序的普及.而MapReduce计算模型能有效地解决上述问题,阐述了Google的MapReduce计算模型的实现机制,并通过实例描述了该模型的执行过程.

In the traditional parallel programming model, how to deal with a large number of data in parallel, how to distribute data for each task, and how to deal with a single point of failure and other issues require a lot of program analysis and design, and the effective treatment of these problems requires programmers to explicitly use the technology. For programmers, this is a very difficult task, because it makes the original simple operation very complex. The existence of these problems, to a certain ex- tent ,restricts the popularity of parallel procedures. The MapReduce calculation model can effectively solve the above problems. This paper describes the implementation mechanism of Google＇s MapReduce calculation model, and describes the implementation process of the model through an example.