Hadoop云平台的一种新的任务调度和监控机制

New Mechanism of Monitoring on Hadoop Cloud Platform

云平台任务监控与资源调度机制是云平台的核心功能之一。Hadoop云平台中任务监控和资源管理的任务是由JobTracker负责处理,并通过slave节点向其发送心跳消息来实现。这种方式导致JobTracker的负载过重,降低了Hadoop云平台的工作效率,限制了Hadoop云平台的规模。提出了一种新的任务监控方案,该方案将JobTracker的任务监控和资源管理功能分离,任务监控功能仍由JobTracker节点完成,资源管理功能由新增的资源管理节点完成,JobTracker通过增量更新的算法将任务调度所需的对象信息动态同步到资源管理节点上,资源管理节点根据心跳消息进行任务分配,并将分配结果返回给JobTracker节点。实验结果表明,本方案不仅通过监控节点实现了任务的监控,增加了监控的灵活性和鲁棒性,而且降低了Jobtracker节点的负担,可有效提高Hadoop云平台的工作效率和规模。

Job monitoring and resource managing mechanism in cloud computing platform are the core functions.The tasks of job monitoring and resource managing in Hadoop are assigned to JobTracker which is implemented on receiving heartbeat message sending by slave node,so JobTracker is the bottleneck of Hadoop.A new mechanism of job monitoring and resource managing scheme was presented.In this scheme,job monitoring and resource managing are separated from original JobTracker,but job monitoring function is still assigned to JobTracker node,and the resource managing function is accomplish by newly added resource monitoring node.JobTracker sends the necessary information of Objects to resource managing node by using a new method of delta updates algorithm.Resource managing node schedules tasks with heartbeat message,and returns the result to JobTracker node.Experimental results show that the scheme implements monitoring node monitor job completely,makes JobTracker node more flexible and robust,reduces the loading of JobTracker node,improves the efficiency of Hadoop platform.