大数据环境下任务调度和资源部署算法研究

Research on Task Scheduling and Resource Deployment in Big Data Environment

为在已有硬件资源配置条件下提高运算效率,提出了基于HDFS分布式存储环境、考虑数据依赖关系的任务调度和资源部署算法。建立的任务调度和资源部署模型包含HDFS集群、数据节点磁盘输入输出负载、网络负载、磁盘输入输出负载的矩阵描述,任务调度和资源部署流程通过设计的任务排程器进行实现。将节点资源评价的函数值作为对算法的评价指标,测试比较了几种算法的性能。在节点资源评价结果方面,与离请求者最近算法、随机算法相比,提出的任务调度和资源部署算法具有较大优势。

In order to improve the operation efficiency under the condition of existing hardware resource configuration,the task scheduling and resource deployment algorithm is proposed,considering data dependencies and distributed storage environment based on HDFS.The task scheduling and resource deployment model includes HDFS cluster,data node disk input and output load,network load,disk input and output matrix description.The process is implemented by task scheduler.Taking the function value of node resource evaluation as the evaluation index of the algorithm,the performance of several algorithms is tested and compared.At last,the experimental results show that the task scheduling and resource deployment algorithm in this paper has more advantages than the nearest algorithm and the stochastic algorithm in resource evaluation.