流式处理框架发展综述

Survey of Distributed Data Flow Processing Framework

大数据已经成为一种重要的信息和知识来源。但是,处理这种大规模的数据有很大的困难,因此衍生出了许多基于大数据的技术平台来进行有效的决策分析。流计算正在引起人们的兴趣,它为实时数据分析提供了机会。因此,大数据处理系统正在朝着更加面向流的方向发展,其中每条数据记录都是通过分布式、低延迟的计算框架在连续的基础上处理的。随着流处理技术的成熟,流分析相关的应用将服务于众多行业。开发流分析基础设施的挑战之一是为不同的应用选择正确的流式处理框架。为了解决这个问题,文章从编程模型、数据分区策略、状态管理、容错恢复等方面,对有代表性的开源(Storm、Spark Streaming、Flink、Kafka Stream)和商业(IBM Streams)流式处理框架进行了全面的对比分析。

Big data has become an important source of information and knowledge.However,dealing with big data of this scale brings great difficulties.Therefore,many technology platforms based on big data have been derived for effective decision analysis.Stream computing is gaining interest and providing opportunities for real-time data analysis.As a result,big data processing systems are becoming more stream-oriented,with each data record processed on a continuous basis through a distributed,low-latency computing framework.As stream processing technology getting matured,stream analysis-related applications will serve many industries.One of the challenges of developing a stream analysis infrastructure is choosing the right stream processing framework for different applications.In order to solve this problem,this paper made a comprehensive comparative analysis of the representative open source(Storm,Spark streaming,Flink,Kafka streaming)and commercial(IBM streaming)distributed stream processing framework from the aspects of programming model,data partitioning strategy,state management,fault tolerance and recovery.