动态在线Map/Reduce流数据处理模型及作业拓扑管理协议

Dynamic Online Map/Reduce Stream Processing Model and Topology Management Protocol

针对海量流数据的在线处理需求,提出一种不同于传统Map/Reduce流数据处理的系统模型Flexible workflow.该模型对workflow处理单元进行在线Map/Reduce并行化,实现了SPATE系统;同时为该系统定义一组关于作业的建立、管理和维护的通信规程,即拓扑管理协议.SPATE系统解决了在线Map/Reduce流数据处理过程中要求实时性及可扩展性的问题.实验验证了拓扑管理协议的有效性,拓扑管理协议能有效管理Flexible workflow流数据处理模型.

To meet the requirements for online processing massive stream data,the authors proposed a novel system model,Flexible workflow,which is different from the traditional Map/Reduce stream data processing.This model conducts the online Map/Reduce parallelization of the process unit of workflow and executes a system of SPATE.A set of topology management protocol was designed for dynamic online Map/Reduce stream data processing model.The protocol includes a group of communication rules about setting up,managing and maintaining jobs.The experimental results validate the topology management protocol is effective,and can manage the Flexible workflow processing model availably.