基于动态带宽分配的Hadoop数据负载均衡方法

Hadoop Data Load Balancing Method Based on Dynamic Bandwidth Allocation

数据负载均衡对Hadoop分布式文件系统(HDFS)性能有着重要的影响,针对HDFS中默认的数据负载均衡方法存在的效率低和缺乏灵活性的不足,文中提出了一种新的动态负载均衡方法,即通过控制变量来动态分配网络带宽以达到数据负载均衡.在此基础上建立了基于控制变量的数据负载均衡数学模型.实验结果表明,文中提出的方法既能保证HDFS的数据访问性能,又能提高集群加入新节点时的数据负载均衡效率.

Data load balancing greatly affects the performance of the Hadoop distributed file system （HDFS）. In order to overcome the inefficiency and inflexibility of the default data load balancing method in HDFS, this paper devises a novel dynamic load balancing method, which dynamically allocates network bandwidth to achieve the data load balancing by controlling variables. Then, the corresponding mathematical model is constructed based on the controlled variables. Experimental results show that the devised method can not only guarantee the performance of the HDFS data access system but also improve the data load balancing efficiency in the presence of a new cluster node.