MapReduce在线抽样分区负载均衡研究

Research on MapReduce On-line Load Balancing Based on Sample Partition

数据倾斜一直是影响MapReduce性能的关键问题之一.为缓解数据倾斜问题,提出一种基于抽样分区的MapReduce在线负载均衡机制:MR-LSP(MapReduce on-line Load balancing mechanism based on Sample Partition).MR-LSP在作业执行之前,通过对源数据抽样分析,预测数据的分布特征,动态采取相应的负载均衡数据分区策略;在作业运行期间实时监控节点负载,进一步动态优化数据分区策略.实验结果表明:MR-LSP能够提高系统3.2%的负载均衡,降低4.3%的作业执行时间,有效缓解了MapReduce的数据倾斜问题.

Data skew has been one of the key issues that affect the performance of MapReduce. To address data skew, the paper proposes a MapReduce on-line load balancing mechanism based on sample partition：MR-LSP. Before job processing,MR-LSP predicts the distribution of data by sampling and analyzing the source data, and dynamically takes appropriate load balancing data partitioning strat- egy ; it can be timely fed back node load information during job running to further optimize the data partitioning strategy. Experimental results show that MR-LSP can improve the load balance of system by 3.2% and reduce job execution time by 4.3% ,and effectively alleviate the MapReduce data skew.