分布式数据流数据倾斜均衡方法研究

Research on Data Skew Equalization Method for Distributed Data Streams

对数据倾斜现象的诱发因素进行了研究,针对分布式数据流的数据倾斜问题提出一种具有实时监控节点负载,及时建立、更新数据迁移路由表的实时的、动态负载均衡方法,通过数据迁移与数据回迁策略保证实现最小迁移代价以及高效的负载均衡。最后模拟实验证明本文提出的负载均衡算法所带来的处理速度上的性能收益远远高于算法自身在系统延迟方面的代价。

The inducing factors of the data skew phenomenon have been studied.In view of the data skew problem of distributed data streams,a real-time,dynamic load balancing method with real-time monitoring of node load and timely establishment and update of data migration routing tables is proposed,in which the minimum migration cost and efficient load balancing are realized by data migration strategies.Finally,the simulation experiment proves that the performance gained in processing speed brought by the load balancing algorithm proposed in this paper are much higher than the cost of the algorithm itself in terms of system delay.