基于蜘蛛猴优化的异构Hadoop任务调度算法

Research on New Optimization Algorithm for Heterogeneous Hadoop Task Scheduling Based on Spider Monkey

针对当前Hadoop在进行任务规划与调度的过程中存在的任务分配不均匀、资源利用率低等问题,考虑Hadoop集群中各节点之间的性能差异,提出一种基于蜘蛛猴优化的异构Hadoop调度算法。首先为了得到每个节点中的任务负载信息,采用集群心跳机制依次获取各节点的内存和CPU信息;然后采用蜘蛛猴优化算法机制构建与任务完成时间相关的目标函数,进而找到任务量与资源分配之间的最优映射关系;最后根据任务类型以及当前集群中个节点的资源利用率,结合最优映射关系完成对对新任务的分配和执行。实验结果表明,与现有的调度算法相比,提出的方法可有效减少任务执行时间、提高调度效率以及任务执行速度。

In response to the problems of uneven task allocation and low resource utilization in current Hadoop task planning and scheduling,and considering the performance differences between nodes in a Hadoop cluster,this paper proposes a heterogeneous Hadoop scheduling algorithm based on spider monkey optimization.Firstly,in order to obtain the task load information of each node,this article adopts a cluster heartbeat mechanism to sequentially obtain the memory and CPU information of each node.Then,the spider monkey optimization algorithm mechanism is used to construct an objective function related to task completion time,and the optimal mapping relationship between task volume and resource allocation is found.Finally,based on the task type and the resource utilization rate of nodes in the current cluster,combined with the optimal mapping relationship,the allocation and execution of new tasks are completed.The experimental results show that compared with existing scheduling algorithms,the method proposed in this paper can effectively reduce task execution time,and improve scheduling efficiency,and task execution speed.