分布式流处理系统中基于位置感知的操作共享优化算法

Operation Sharing Optimization Algorithm Based on Location Aware in DSPS

针对流数据规模大、基本数据处理操作有重叠等特点,提出一种基于位置感知的操作共享优化算法.假设操作预先分配了计算节点资源,在保证流处理实时性和动态性的基础上,利用多个流数据作业间拓扑结构和操作功能的相似性,基于DAG图匹配方法建立操作共享备选集;以输出带宽为约束条件,建立一个操作共享收益优化问题模型,解决了流处理过程中传统操作共享方法对集群的输出带宽影响考虑不充分的问题.仿真实验结果表明,该方法较原有算法更充分地考虑了计算节点制约因素,并提高了共享收益,从而可有效地节省系统计算资源.

We proposed an operation sharing optimization algorithm based on the location aware according to the data flow in large scale,and the basic data processing operations had overlapping features.We assumed that the operations had assigned the resources of compute nodes in advance,in guaranteed the stream processing real-time and dynamic state,to take advantage of the similarity between topologies and operational functions of multiple stream data operations.We established an operation sharing revenue optimized model based on the DAG matching method on the constraint of output bandwidth.This method tackled the insufficiency consideration of traditional operation for operation sharing method to the cluster of output bandwidth.Simulation experiment results show that this method has fuller consideration in the calculation of the node constraints and improves the benefits of sharing,thereby saving the computing resources of the system effectively.