面向大规模感知数据的实时数据流处理方法及关键技术

Real-time data stream processing and key techniques oriented to large-scale sensor data

为了在大规模历史感知数据基础上实现针对高速传感数据流的实时计算,提出一种面向大规模历史数据的数据流处理方法RTMR,通过中间结果缓存、流水化和本地化改进了MapReduce的数据流处理能力。在此基础上,为了适应性地构造RTMR集群,利用模型分析方法根据应用特征和集群环境配置节点类型和拓扑结构。为实现集群的负载均衡,通过计算负载状态转换关系分组空闲节点和过载节点,将NP难的动态负载均衡问题快速分解为规模较小的子问题,并且综合执行时间和数据移动代价作为子问题的优化目标,提高应对负载倾斜的反应速度。实验表明,上述方法和技术能够保障大规模历史数据上数据流处理的可伸缩性。

With the development of Internet of Things, how to realize real time computation for high speed data stream based on large scale history sensor data became a new challenge to cloud manufacturing. A processing meth- od named Real-Time MapReduce （RTMR） oriented to large scale historical data was proposed, which improved data stream processing capacity of MapReduce through intermediate result cache, pipelining and localization. To con- struct RTMR sets, the model analysis method was used to configure the node type and topological structure based on application characteristics and cluster environments. Furthermore, to realize cluster load balancing, the idle nodes and overload nodes were grouped by computing load state transition relation. Thus the dynamic load balancing problem of NP hard was decomposed into small scale sub-problems, and execution time as well as data cost were in- tegrated as sub-problem＇s optimization objective. The experiment result showed that the proposed method and tech- nology could ensure the scalability for data stream processing of large scale historical data.