基于学习的容器环境Spark性能监控与分析

Learning-based performance monitoring and analysis for Spark in container environments

Spark计算框架被越来越多的企业用作大数据分析的框架,由于通常部署在分布式和云环境中因此增加了该系统的复杂性,对Spark框架的性能进行监控并查找导致性能下降的作业向来是非常困难的问题。针对此问题,提出并编写了一种针对分布式容器环境中Spark性能的实时监控与分析方法。首先,通过在Spark中植入代码和监控Docker容器中的API文件获取并整合了作业运行时资源消耗信息;然后,基于Spark作业历史信息,训练了高斯混合模型(GMM);最后,使用训练后的模型对Spark作业的运行时资源消耗信息进行分类并找出导致性能下降的作业。实验结果表明,所提方法能检测出90.2%的异常作业,且其对Spark作业性能的影响仅有4.7%。该方法能减轻查错的工作量,帮助用户更快地发现Spark的异常作业。

The Spark computing framework has been adopted as the framework for big data analysis by an increasing number of enterprises. However, the complexity of the system is increased due to the characteristic that it is typically deployed in distributed and cloud environments. Therefore, it is always considered to be difficult to monitor the performance of the Spark framework and finding jobs that lead to performance degradation. In order to solve this problem, a real-time monitoring and analysis method for Spark performance in distributed container environment was proposed and compiled. Firstly, the resource consumption information of jobs at runtime was acquired and integrated through the implantation of code in Spark and monitoring of Application Program Interface （API） files in Docker containers. Then, the Ganssian Mixture Model （GMM） was trained based on job history information of Spark. Finally, the trained model was used to classify the resource consumption information of Spark jobs at runtime and find jobs that led to performance degradation. The experimental results show that, the proposed method can detect 90.2% of the abnormal jobs and it only introduces 4.7% degradation to the performance of Spark jobs. The proposde method can lighten the burden of error checking and help users find the abnormal jobs of Spark in a shorter time.