分布式流处理系统的时间感知分组算法

Time-aware grouping algorithm for distributed stream processing systems

随着对实时数据流处理需求的增加,分布式流处理系统的发展也越来越受到关注。大量的倾斜的数据流以及复杂分布式系统的异构性对当前的分布式流处理系统的分组策略提出了挑战。目前已有的分布式流处理分组策略通常关注并行实例之间元组数量的均衡性,而忽视了系统异构性对分组策略造成的影响。该文提出了一种时间感知分组算法,通过对分布式流处理系统存在的网络异构性和处理能力异构性的分析,综合考虑流处理系统中各下游算子实例的处理时间以及上游算子与下游算子之间的通信时间,并根据键值的频率不同制定不同的路由策略,在较小的开销下使系统达到负载均衡。在Apache Flink分布式流处理系统上进行的实验结果表明:时间感知分组算法比已有的分组算法在系统吞吐量上提高了10%,在平均处理延迟上降低了33%。

The increasing demand for real-time data stream processing is driving the development of distributed stream processing systems.The large amount of skew data streams and the heterogeneity of complex distributed systems pose challenges to the current grouping strategies of distributed stream processing systems.The existing distributed stream processing grouping strategies usually focus on balancing the number of tuples between parallel instances,while ignoring the impact of system heterogeneity on the grouping strategy.This paper presents a time-aware grouping algorithm that analyzes the network heterogeneity and the processing capability in a distributed stream processing system that considers the processing time of each downstream operator instance in the stream processing system.The algorithm also takes into account the communication time between the upstream and downstream operators with various routing strategies formulated according to the frequency of the key,so that the system achieves load balancing with little overhead.Tests on an Apache Flink distributed stream processing system show that the time-aware grouping algorithm increases the throughput by 10%while the average processing latency is reduced by 33%compared to the existing grouping algorithm.