采用资源划分的云环境下Hadoop资源许可调度方法

A Resource License Scheduling Method for Hadoop in Cloud Computing Using Resource Allocation

针对云计算环境中Hadoop平台由于节点计算能力差异、多样混合负载共存等原因而出现的性能不佳的问题,提出一种采用资源划分的资源许可方法。该方法在云计算环境下通过减少资源浪费或负载过重等情况的出现来提高系统性能。该方法采集资源信息并推测任务资源需求,根据可用计算资源和任务需求动态划分、调度资源;使用与资源无耦合的资源许可启动任务并控制任务数量调节资源利用率以适应云环境。使用该方法对比公平调度器在国家高性能计算中心(西安)进行测试发现:单作业在资源竞争环境中优于公平调度器的静态结果;混合负载在3种测试环境中完成时间分别平均减少了27.5%、37.1%和50.98%,性能显著提升。实验结果表明,该方法可以适应负载资源需求和可用计算资源的变化,灵活划分计算资源,解决Hadoop在云环境中的性能不佳问题。

A resource license scheduling method using resource allocation is proposed to improve the poor performance caused by different computing capacities of nodes and mixed workload in cloud computing. The method improves the performance through reducing the resources wasting or overload. It collects resource information and estimates resource requirements of workloads and allocates computing resources dynamically according to available resources and resource requirements of workloads. Licenses uncoupled with resources are used to launch tasks and to adjust the number of parallel tasks to adapt the cloud environment by controlling the number of licenses. The method is evaluated in the national high performance computing center （Xi＇an）. Results show that the completion times of single job workloads of the proposed method are better than those of the FAIR scheduler in competitive environments. Moreover, the completion times of mixed workloads of the method in three environments reduce 27.5%, 37. 1% and 50.98% respectively on average, that is, the performance of the method has a significant improvement. It can be concluded from the results that the proposed method adapts the complex environment and solve the performance problem in cloud computing.