基于Flink的任务调度策略

Task scheduling strategy based on Flink environment

针对大数据流式计算平台拓扑中因各关键节点上任务间不同类型的通信方式导致的通信开销较大问题,提出一种Flink环境下的任务调度策略。通过各任务间数据流大小确定拓扑边权重,将有向无环图转化为拓扑关键路径模型,在保证关键路径上节点负载差异较小的同时,最小化关键任务的节点间通信开销。实验结果表明,该算法与Flink平台现有的任务调度策略相比,在WordCount和TwitterSentiment作业执行过程中计算平均时延降低了13.09%,有效提升了系统性能。

Focusing on reducing the large communication overhead caused by different communication types among tasks on key nodes in the big data streaming platform topology,to alleviate this problem,a task scheduling strategy in Flink environment was proposed.The weight of topological edges was determined by the size of data flow among individual tasks,and the directed acyclic graph was transformed into a topological critical path model.The inter-node communication overhead of critical tasks was minimized while ensuring little difference of node load on the critical path.Experimental results show that compared with the existing task scheduling strategies of Flink platform,the proposed strategy can improve the performance of system by an average of 13.09%when calculating latencies of the WordCount and TwitterSentiment operations.