面向大规模数据接入系统的负载平衡机制

Load balancing mechanism for large-scale data access system

当前分布式系统负载平衡算法存在问题:1)算法建立的系统中各节点角色固定,系统不具有自适应性;2)算法的通用性不高;3)负载迁移任务巨大,且负载平衡周期过长等。针对这些问题,提出了混合式负载平衡算法。首先,设计了一个分布式系统接收模型。模型将系统任务分为三层:接收层、处理层和存储层。在接收层使用了自定义的通信协议提高系统的接收性能。然后,负载平衡算法采用随机负载迁移策略,根据系统中节点的负载状态,对负载任务进行随机迁移。通过这种策略解决负载平衡周期过长和负载回迁问题。最后,通过分布式控制节点选择策略,使系统中节点具有自适应性。实验结果显示,在百万数据源以下,系统各层平均延迟处于毫秒级,系统负载平衡平均耗时在3 min以下。实验证明了所提出的负载平衡机制具有周期短、任务响应迅速等特点,能够提高分布式系统的接收性能。

Some problems of the current load balancing algorithms for distributed systems include： 1） The role of each node in the system is fixed, and the system has no adaptability. 2） The load balancing algorithm is not universal. 3） The migration task is too large, and the load balance cycle is too long. To solve these problems, a hybrid load balancing algorithm was proposed. Firstly, a distributed receiving system model was designed, by which the system tasks were divided into three parts： receiving level, handling level and storing level. In receiving level, a home-made transmission protocol was used to improve the reception capability of the system. And then, in the load balancing algorithm, random load migration strategy was used. According to the status of the nodes, the tasks of load were randomly migrated. The problems of long load balance cycle and load moving back were solved by this strategy. Finally, the distributed control node selecting strategy was adopted to make the nodes adaptable. The experimental results show that the average delay in each layer of the system is in milliseconds, and the system load balancing takes less than 3 minutes, which proves that the load balancing mechanism has short load balance cycle and fast response, and can improve the reception capability of the distributed system.