广播机制解决Shuffle过程数据倾斜的方法

Method Research to Solve Shuffle Data Skew Based on Broadcast

在Spark 计算平台中,数据倾斜往往导致某些节点承受更大的网络流量和计算压力,给集群的CPU、内存、磁盘和流量带来了巨大的负担,影响整个集群的计算性能.本文通过对Spark Shuffle 设计和算法实现的研究,深入分析在大规模分布式环境下发生数据倾斜的本质原因.提出了广播机制避免Shuffle 过程数据倾斜的方法,分析了广播变量分发逻辑过程,给出广播变量性能优势分析和该方法的算法实现.通过Broadcast Join 实验验证了该方法在性能上有稳定的提升.

In the Spark computing platform, data skew often causes some nodes to withstand greater network traffic and computing pressure, which imposes a huge burden on the cluster’s CPU, memory, disk, and traffic, affecting the computing performance of the entire cluster. Through the research on Spark Shuffle design and algorithm implementation, and deep analyses on the essential reasons of data skew in large-scale distributed environment, this study proposes a method to avoid data skew in shuffle process through the broadcast mechanism, analyzes the process of broadcast variable distribution logic, and gives the algorithm implementation and performance advantage analysis of the method. The performance of the method is improved by the Broadcast Join experiment.