分布式实时抽取计算框架设计与应用

Design and application of distributed real time extraction calculation framework

针对任务类型的多样化、大量化、复杂化带来的数据源结构、数据类型的问题,文中提出了将Zookeeper消息调度和Redis内存计算相结合的方式,设计分布式实时抽取框架。利用Zookeeper消息通知,队列优先,共享锁等功能,结合Redis支持网络、可基于内存计算亦可持久化日志存储等特性,系统的吞吐量、执行效率、存储效率等都有显著提高。

Due to data source and type challenges which caused by tasks of different varieties,sizes and complexities,this paper proposes to combine the Zookeeper message scheduling with the Redis memory computing,to design a distributed real time extraction framework. Using Zookeeper message notification,priority queue,shared locks and other functions,with Redis supporting network,based on the calculation of memory and persistent log storage and other characteristics,the throughput,efficiency and storage efficiency of system are significantly improved.