基于垃圾回收的MapReduce作业内存调优

GC-based MapReduce Job Memory Tuning

针对合理管理MapReduce作业内存资源困难的问题,提出评估方法并给出优化配置建议。首先分析Java虚拟机的内存分配与垃圾回收的原理,给出垃圾回收重要指标;其次提出内存分配合理性评估的3种指标和评估方法;最后根据评估结果给出2种优化配置建议:一是通过使用聚类算法和统计信息来估计晋升对象大小阈值,优化Java虚拟机的对象分配和垃圾回收性能;二是使用回归模型和搜索算法来预测作业合理的内存配置。实验结果表明,提出的方法能自动发现作业内存配置的不足并给出优化的配置建议。与采用机器学习方法相比,提出的方法不需要运行大量的测试,因此该方法能很好适用于MapReduce的生产集群环境。

Different Job requires different memory resources,and it is difficult to assess the rationality for a memory allocation to a MapReduce Job. In order to solve this problem,an assessment method was presented and recommended for memory settings of JVM where Job＇s tasks run. Firstly,some important GC metrics were introduced based on the analysis of JVM＇s memory allocation and GC workflow in-depth. Then,three kinds of indicators and memory allocation rationality evaluation method were introduced based on the three indicators. Finally,two kinds of optimal JVM configuration were recommended,which are using K-means algorithm and statistical information to estimate the threshold value of the object size which should have been allocate in old generation,and modeling GC pause time and using search algorithm to predict the size of young generation and the old generation,respectively. Experimental results showed that the proposed approach can automatically find insufficient of memory configuration of a Job. Compared with using machine learning methods,the proposed method does not need to run a large number of test cases,so it can apply to production cluster of MapReduce.