摘要
MapReduce is currently the most popular programming model for big data processing, and Hadoop is a weU-known MapReduce implementation platform. However, Hadoop jobs suffer from imbalanced workloads during the reduce phase and inefficiently utilize the available computing and network resources. In some cases, these problems lead to serious performance degradation in MapReduce jobs. To resolve these problems, in this paper, we propose two algorithms, the Locality-Based Balanced Schedule （LBBS） and Overlapping-Based Resource Utilization （OBRU）, that optimize the Locality-Enhanced Load Balance （LELB） and the Map, Local reduce, Shuffle, and final Reduce （MLSR） phases. The LBBS collects partition information from input data during the map phase and generates balanced schedule plans for the reduce phase. OBRU is responsible for using computing and network resources efficiently by overlapping the local reduce, shuffle, and final reduce phases. Experimental results show that the LBBS and OBRU algorithms yield significant improvements in load balancing. When LBBS and OBRU are applied, job performance increases by 15% from that of models using LELB and MLSR.
MapReduce is currently the most popular programming model for big data processing, and Hadoop is a weU-known MapReduce implementation platform. However, Hadoop jobs suffer from imbalanced workloads during the reduce phase and inefficiently utilize the available computing and network resources. In some cases, these problems lead to serious performance degradation in MapReduce jobs. To resolve these problems, in this paper, we propose two algorithms, the Locality-Based Balanced Schedule （LBBS） and Overlapping-Based Resource Utilization （OBRU）, that optimize the Locality-Enhanced Load Balance （LELB） and the Map, Local reduce, Shuffle, and final Reduce （MLSR） phases. The LBBS collects partition information from input data during the map phase and generates balanced schedule plans for the reduce phase. OBRU is responsible for using computing and network resources efficiently by overlapping the local reduce, shuffle, and final reduce phases. Experimental results show that the LBBS and OBRU algorithms yield significant improvements in load balancing. When LBBS and OBRU are applied, job performance increases by 15% from that of models using LELB and MLSR.
基金
supported by the National Key R&D Program of China(Nos.2017YFB0202104 and 2017YFB0202003)
