基于负载预测的HDFS动态负载均衡改进算法

Improved HDFS Dynamic Load Balancing Algorithm Based on Load Prediction

针对现有HDFS负载均衡改进算法在解决负载状态衡量的片面性、滞后性以及阈值设定的静态性问题等方面存在不足,提出了一种基于负载预测的HDFS动态负载均衡改进算法。该算法首先探究影响存储效率的负载特征,定义多指标负载计算数学函数,然后通过优化的二次指数平滑预测算法,对节点下一时刻的负载值进行动态预测,最后根据预测结果和节点性能对集群的繁忙程度、响应效率和均衡程度等信息进行综合评估,建立动态阈值计算模型,进而对集群做出均衡判断和决策,更好地实现集群动态均衡效果。理论分析和实验结果表明,改进算法对存储系统中的负载均衡调度具有高效性,在达到更好均衡效果的同时也缩短了作业的完成时间,提高了系统整体响应效率。

The existing HDFS load balancing improvement algorithm has some shortcomings in solving the one-sidedness,hysteresis,and static setting of the threshold state.An improved HDFS dynamic load balancing algorithm based on load prediction is proposed.Firstly,the algorithm explores the load characteristics that affect storage efficiency,defines the mathematical function of multi-index load calculation,and then dynamically predicts the load value of the node at the next moment through the optimized quadratic exponential smoothing prediction algorithm.Finally,based on the prediction result and node performance.The information such as the busyness,response efficiency and balance degree of the cluster is comprehensively evaluated,and a dynamic threshold calculation model is established,thereby making balanced judgments and decisions for the cluster,and better realizing the dynamic balancing effect of the cluster.Theoretical analysis and experiments show that the improved algorithm is efficient for load balancing scheduling in the storage system.While achieving better balancing effect,it also shortens the completion time of the operation and improves the overall response efficiency of the system.