基于共享缓冲区的数据流处理框架设计与实现

Design and implementation of data stream processing framework based on shared buffer

为解决数据流处理应用程序依赖数据流传输,分布式环境部署困难,应用开发难度大等问题,设计了一种基于共享缓冲区的数据流处理框架。该框架利用共享缓冲区为应用提供数据代理接口,实现数据流推动和分布式处理,并引入可自定义的负载均衡,解决了应用对数据流的依赖度和开发部署的难度系数,使其专注数据分析处理,无需关心数据的地理位置,提高了应用系统的稳定性和扩展性。通过在互联网病毒监控系统中应用,验证了该框架在分布式环境下的实时性和灵活性,使其可移植到云计算环境中。

To solve the data transfer dependence and to decrease coefficient of difficulty in the deployment of the distributed environment and the development, a data stream processing framework base on shared buffer is proposed for data process processing application. This design using a shared buffer proxy provides data for application, push data among nodes and process in distributed manner. User define load balancing is introduced. A data dependence for application is solved, and makes it realized data analysis and processing, without concern for the geographic location of the data. The stability and scalability of the application system is improved. Finally, the real-time and flexibility of the framework in a distributed environment is validated by realizing the Internet virus monitoring system. So that it can be ported to a cloud computing environment.