基于蚁群算法的Storm集群资源感知任务调度

Research on Storm Resource-aware Task Scheduling with Ant Colony Algorithm

实时计算系统Storm是当前十分流行的开源流式系统,在处理流式数据时具有明显的优势,但也存在默认调度器在任务调度时难以将节点资源与任务需求相结合、节点资源利用率不高、节点内存不足以及网络堵塞等问题。为了解决这些问题,提出了一种基于蚁群算法的Storm集群资源感知任务调度算法及其实现方案。该算法将节点的资源动态变化表示为蚂蚁运动所需的信息素,将任务调度过程模拟为蚂蚁觅食过程,以此对任务调度进行优化,保证了Storm任务调度的有效性。实验结果表明,该算法能够找到与当前任务所需资源最匹配的节点,从而实现资源的合理分配;与默认调度相比,具有更优的任务调度效率、更少的平均处理时间和更高的集群吞吐量,有利于集群负载均衡,优化集群的性能。

Storm is the popular open source real-time computing system, which has a great advantage in handling data stream. However, there are some problems in its default scheduler when scheduling tasks, such as difficultly in combining node resources and mission re- quirements, ineffectiveness in node resource utilization, lack of memory and network congestion and so on. In order to solve them, a re- source-aware scheduler based on ant colony algorithm and its implementation scheme is proposed, in which the dynamic changes of the node resource can be expressed as the pheromones of ant movement required and the task scheduling process is similar to ant foraging process, to optimize the task scheduling and ensure the effectiveness of the Storm task scheduling. Experimental results show that it has found the most suitable node for the current task and achieved the reasonable allocation of resources and that compared with the default scheduling, it has better task scheduling efficiency, less average processing time and higher throughput of the cluster, which can benefit the load balance and optimize the performance for the cluster.