异构Hadoop集群中基于节点特性的负载平衡方法

Research on Load Balancing Method Based on Node Characteristics in Heterogeneous Hadoop Clusters

为了解决异构Hadoop集群中每个结点的负载平衡问题,提出了一种基于节点特征的负载均衡方法。方法首先分析决定节点性能的主要指标,然后根据这些指标的贡献,定义了对应的目标函数来描述节点性能;同时将节点性能与节点忙闲状态相结合来计算集群中每个结点的相对负载值。通过分析每个节点的相对负载值和集群存储利用率,计算每个节点的存储利用率的推荐值。最后,基于当前集群的磁盘负载状态,动态生成平衡阈值。实验结果表明,提出的负载均衡方法可为异构Hadoop集群提供更合理的均衡方案,同时可以提高运行效率和缩短了执行时间。

To solve the load balancing problem of each node in heterogeneous Hadoop clusters,a node feature-based load balancing method is proposed.This method first analyzes the main indicators that determine the node performance.Then,according to the contribution of these indicators,the corresponding objective function is defined to describe the node performance;At the same time,the relative load value of each node in the cluster is calculated by combining the node performance with the node busy state.The recommended value of storage utilization of each node is calculated by analyzing the relative load value of each node and the cluster storage utilization.Finally,the balance threshold is dynamically generated based on the current disk load status of the cluster.The experimental results show that the load balancing method proposed in this paper can provide a more reasonable balancing scheme for heterogeneous Hadoop clusters,and can improve the running efficiency and shorten the execution time.