一种有效的 Hadoop 参数优化模型

AN EFFICIENT PARAMETER OPTIMIZATION STRATEGY OF HADOOP

为了使 Hadoop 集群能够应对复杂多变的作业，减少作业类型差异对集群性能所带来的影响，提出了一种参数优化模型实现对集群参数的自动调优配置。参数优化模型会根据作业类型及输入数据的规模选取相应的参数组合进行自动优化，然后通过改进的和声算法迭代产生最优的参数配置。实验结果表明，参数优化模型的自动调优保证了集群工作性能的充分发挥，有效的缩短了集群执行作业的运行时间，使集群具有良好的稳定性和扩展性。

In order to make the Hadoop cluster to cope with the complex jobs,reduce the impact of cluster performance caused by job type differences,this paper proposes a parameter optimization model to automatically tune the configuration of cluster parameters. The parameter optimization model is based on job types and the size of the input data to select the corresponding parameter combination to automatically optimize,and then generate the optimal parameter configuration through the improved harmony algorithm. Experiment results show that the automatic tuning of parameter optimization model guarantees full play to the cluster performance,shortens the working time of cluster tasks effectively,and makes the cluster has a good stability and scalability.