分布式流处理系统的容错性能基准测试

Benchmarking for Fault-tolerant Performance in Distributed Stream Processing Systems

随着对数据处理的实时性要求越来越高,分布式流处理系统应运而生。但是在分布式的集群规模下,各种软硬件原因导致的故障很难避免的。现有的相关基准测试主要关注于分布式流处理系统的处理性能,很少对该类系统处理故障的容错性能进行评测,以至于关键应用在系统选型的时候特别艰难。针对分布式流处理系统的容错性能,本文设计并实现了一套灵活的基准测试框架。最后,本文在开源数据流处理系统Apache Storm和Apache Flink进行了容错性能的基准测试,验证定义的测试基准的正确性和有效性,实验结果也表明Flink的容错性能相对较好。

With the increasing real-time requirements for data processing,distributed stream processing systems have emerged.However,under the distributed cluster scale,failures caused by various hardware and software problems are inevitable.The existing related benchmarking mainly focus on the performance of the distributed stream processing system during failure-free time,while rarely evaluating the fault-tolerant performance of the system for handling faults.As a result,it is particularly difficult to select a system for mission-critical applications.This paper designs and implements a flexible benchmarking framework tailored for fault-tolerant performance.Finally,benchmarking the fault-tolerant performance of Apache Storm and Apache Flink verifies the correctness and effectiveness of the benchmark defined in this paper.Experimental results show that fault-tolerant performance of Flink outperforms that of Storm.