Linux内核参数对Spark负载性能影响的研究

Impact of Linux kernel parameters on Spark workloads

关于Spark性能的研究目前正在成为热点,但调优策略多位于应用层,而不是系统层。操作系统作为硬件之上的第一层软件,对硬件性能发挥起着根本作用。Linux内核提供了丰富的参数作为优化性能的接口,但实际中,这些参数的作用并没有得到充分发挥。人们更多是采用系统默认值,而不是根据具体环境进行调整。然而本文实验发现,系统默认值并不一定是最好的选择,有时甚至是最坏的。定义了"影响比"这一概念,并基于此概念提出了一种通过分析内核函数的执行情况来认识参数对Spark应用影响的方法。针对Spark内存计算的特点,从大页、NUMA这两个与使用内存紧密相关的方面分析了相关内核参数对几种典型Spark负载的性能影响,并由此得出一些结论。希望本文的分析和结论可以为Spark平台合理设置内核参数提供一些参考。

Research on the performance of Spark becomes a hot topic, however, optimization strategies are mostly used on the application level instead of system level. As the first software above hard- ware, the operating system plays a fundamental role in the performance of hardware. The Linux kernel provides abundant parameters as the interface to optimize the performance of the system. However, in practice, kernel parameters have not fully played their roles. Most people use their default values rather than change them to fit the specific environment. However, our experiments prove that the default values are not always the best choice, and sometimes it is even the worst. We define the concept of ＂influ- ence ratio＂, and put forward a method based on the concept to understand the influence of parameters on Spark applications by analyzing the kernel functions. According to the features of the memory computing of Spark, we analyze the influence of Linux kernel parameters on several typical Spark workloads from the aspects of Transparent Huge Page and NUMA, which closely relates to the use of memory, and then give some conclusions. We hope that the analysis and conclusions can provide some experience of tuning kernel parameters reasonably for the Spark platform.