基于多阶段划分的MapReduce模型

MapReduce performance model based on multi-phase dividing

针对已有的MapReduce模型阶段划分粒度不合理导致模型精度和复杂度存在的问题,提出了阶段划分粒度为5的多阶段MapReduce模型（MR-Model）。首先综述了MapReduce模型的研究现状;然后将MapReduce划分为Read、Map、Shuffle、Reduce、Write共5个阶段,并对每个阶段的具体运行时间进行研究;最后通过实验对模型的预测性能进行验证。实验结果表明,提出的MR-Model可用来描述MapReduce实际任务的执行过程,与另外两种不同划分粒度的模型P-Model和H-Model相比,MR-Model模型的运行时间预测精度可以提高10%-30%,在Reduce阶段的运行时间预测精度可以提高2-3倍,综合性能较好。

In order to resolve the low precision and complexity problem of the existing MapReduce model caused by the reasonable phase partitioning granularity, a multi-phase MapReduce Model（ MR-Model） with 5 partition granularities was proposed. Firstly, the research status of MapReduce model was reviewed. Secondly, the MapReduce job was divided into 5phases of Read, Map, Shuffle, Reduce, Write and the specific processing time of each phase was studied. Finally, the MRmodel prediction performance was tested by experiments. The experimental results show that MR-Model is suitable for the MapReduce actual job execution process. Compared with the two existing models of P-Model and H-Model, the time accuracy precision of MR-Model can be improved by 10%- 30%; in the Reduce phase, its time accuracy precision can be improved by2- 3 times, the comprehensive property of the MR-Model is better.