实时流处理系统Storm的调度优化综述

Survey of real-time processing system Storm scheduling optimization

随着大数据技术的发展,相对于Hadoop等传统的批处理系统,流式处理系统具有更好的实时性特点。在已有的流式处理系统中,Storm系统具有良好的稳定性、高可扩展性以及高容错性等优点,使它在流式数据处理系统中脱颖而出。但是在任务调度方面,Storm系统并没有做过多的考虑,默认采用相对简单的轮询调度法,导致系统在性能上存在瓶颈。近年来针对Storm系统的调度问题,研究提出了各种优化方案。从实时流处理系统Storm的调度优化出发,将这些优化方法分为四类,并详细阐述各类中具有一定代表性的方法,分析其优缺点以及适用的场景。最后,讨论了在日益发展的新环境下,Storm系统的调度优化相关研究未来可能存在的方向。

With the development of large data technology,compared with Hadoop and other traditional batch system,streaming processing system has better real-time characteristics.In the existing flow processing system,the Storm system has the advantages of good stability,high scalability and high fault tolerance,so that it can stand out in the flow data processing system.How-ever,in the task scheduling,Storm system has not done too much consideration,the default using a relatively simple polling scheduling method,resulting in the performance bottleneck in the system.In recent years,this paper proposed a variety of optimization schemes have been for the scheduling problem of Storm system.Based on the scheduling optimization of real-time stream processing system Storm,this paper divided the method into four categories,and described the methods of representative representation in each category in detail,and analyzed their advantages and disadvantages.Finally,it discussed the possible future direction of Storm system scheduling optimization in the new environment.