一种基于Value均值的MapReduce任务分配策略

A Allocation Strategy for MapReduce Task Based on Value-Mean

在大数据处理中,MapReduce编程思想是处理海量数据中值得借鉴的思想,其计算任务可分为Map任务与Reduce任务。不同类型的数据,其来源与格式不同,处理时不同Key值的List<Value>集合不同,导致不同Reduce任务节点负载不同,体现在集群上为各个节点任务负载不均衡。针对MapReduce中不同Reduce任务节点负载不均衡问题,在Reduce任务中,将同一Key值的List<Value>集合进行均值处理,并根据Key值进行重新划分,提出一种基于Value均值的MapReduce任务分配策略。实验证明,该策略不仅能够提高Reduce任务处理效率,而且具有广泛性。

In large data processing, MapReduce programming is a valuable idea for dealing with massive data. Its computing tasks can be divided into Map tasks and Reduce tasks. Different types of data have different sources and formats. Different sets of List < Value> with different Key values are processed differently, resulting in different Reduce task nodes with different load, which is reflected in the uneven task load of each node on the cluster. Aiming at the problem of unbalanced load among different Reduce task nodes in MapReduce task, this paper proposes a MapReduce task allocation strategy based on Value Mean by means of the same Key value List < Value > set and re-partitioning according to Key value. Experiments show that the strategy can not only improve the efficiency of Reduce task processing, but also has good universality.