MapReduce中连接负载均衡优化研究

Optimizing load balancing of joins in MapReduce

数据分析和处理是大规模分布式数据处理应用中的重要任务。由于简单易用和具有灵活性,MapReduce编程模型逐渐成为大规模分布式数据处理系统(如Hadoop系统)的核心模型。由于所处理的数据可能不是均匀分布的,MapReduce编程模型在处理连接操作时,会出现数据倾斜问题。数据倾斜问题严重降低了MapReduce执行连接操作的效率。针对MapReduce中连接操作的数据倾斜问题,分析了造成MapReduce连接性能瓶颈的原因并建立负载均衡代价模型,提出了用范围分割方法控制连接过程中的数据倾斜问题实现负载均衡的策略。实验结果表明,所提方法明显提高了连接的效率。

Data analysis and processing is one of the most important tasks in large-scale distributed data processing applications. Due to its simplicity and scalability, MapReduce programming model has gradually become the crucial model for large-scale distributed data processing systems （eg. Hadoop）. Since the data may be uniformly distributed, data skew occurs when MapReduce programming model joins data,thus degrading the join performance severely. To solve data skew, its reason is analyzed, the load balancing cost model is established, and the rangepartitioner algorithm is proposed to control data skew so as to realize load balancing. Experimental results demonstrate that our method can obviously im- prove the efficiency of joins.