Hadoop集群性能参数自动调优信息库系统构建

Hadoop Cluster Performance Parameters Auto Tuning Information Library System Build

为了使Hadoop集群系统能够应对多变的任务及系统本身节点差异对集群性能带来的影响,采用TaskConfigure服务器构建Hadoop集群参数信息库系统实现对集群参数的自动调优配置.通过对集群节点及任务的分类,提出集群按类分配配置参数及采用节点资源利用效率生成集群系统参数的优化配置值.实验结果表明,参数信息库系统的自动调优保证了集群工作性能的充分发挥,有效地缩短了集群执行任务的工作时间,使集群具有良好的稳定性和扩展性.

To make Hadoop cluster system to cope with the impact of varied tasks and cluster performance caused by system node differences, building Hadoop cluster parameters information library system on TaskConfigure server to automatically tune configure cluster parameters. Adopt of the classification of the cluster nodes and tasks, proposing that cluster configuration parameters assigned by category and cluster optimized configuration values of the system parameters generated by nodes efficiency of resource using. Ex- periment results show that the automatic tuning of information library system guarantee full play to the cluster performance of work, shorten the working time of cluster tasks effectively , make the cluster has a good stability and scalability.