Heron环境下基于实例重分配的传输负载优化策略

Transmission load optimization strategy based on instance reallocation in Heron

作为新一代大数据流式计算框架,Heron忽略了任务实例之间不同通信方式的差异以及节点资源利用率不均衡的问题导致系统性能下降。针对这一问题,设计了节点资源限制模型、通信开销优化模型和实例数据流关系模型,并在此基础上提出了Heron环境下基于实例重分配的传输负载优化策略(transmission load optimization strategy based on instance reallocation in Heron,TLIR-Heron)。该策略包括节点资源限制算法和实例重分配算法,通过判定实例重分配条件并执行重分配算法将节点间数据流转换为节点内数据流,从而降低通信开销。实验结果表明,在三组拓扑测试下,TLIR-Heron相较于Heron默认调度策略能够降低节点间通信开销和系统的计算延迟,并提升了计算节点资源利用的均衡性。

As a new platform in big data stream computing,Apache Heron ignores the difference in communication modes between task instances and the unbalance of processing load among nodes,which leads to the decline system performance.To address the problem,this paper designed the model of node resource limitation,the model of communication overhead optimization and the model of data stream relationships among instances,as the foundation to propose the TLIR-Heron.The strategy was composed of the node resource limitation algorithm and the instance reallocation algorithm.By judging the criteria for instance reallocation and executing instance reallocation algorithm,this strategy transformed the inter-node data streams into intra-node data streams and minimized the communication overhead of the system.The experimental results show that under the three sets of benchmarks,TLIR-Heron reduces the communication overhead between nodes and the response latency of the system compared with the default scheduling strategy,and improves the balance of resource utilization of computing nodes.