电力全业务统一数据中心突发性数据处理任务调度方法

Scheduling Method for Burst Data Processing Task in Full-service Unified Data Center of Electric Power System

国家电网有限公司正在开展电力全业务统一数据中心的建设,以实现全业务范围、全数据类型、全时间维度数据的统一存储、管理分析。在全业务范围数据全量推送过程中,尤其在突发性海量数据涌入时,全业务统一数据中心的处理能力面临着巨大挑战,数据处理任务调度方法是提高其数据处理能力的关键。通过对电力全业务统一数据中心设计方案和架构的分析,给出一种数据处理任务控制调度模型,从任务的时间和有效价值两方面综合分析任务的优先级。提出了基于多级队列的动态实时优先级调度(DRPS)算法,并分析了不同类型处理任务的优先级动态变化趋势。采用CloudSim平台模拟全业务统一数据中心的数据抽取—转换—加载(ETL)任务,并进行仿真测试。结果表明DRPS算法相比于动态价值密度(DVD)算法及最早截止优先(EDF)算法,在截止期错失率方面有明显下降,任务完成率也有明显提高。

State Grid Corporation of China is now carrying out the construction of the full-service unified data center of electric power system to achieve the unified storage and management analysis of the data in the full-service scope,all data types and all time dimensions.While pushing the full-service business data,especially during the influx of burst mass data,the processing capacity of the full-service unified data center will face a great challenge.The scheduling method of data processing task is vital to improve the data processing capacity of the data center.Based on the analysis of the design scheme and architecture for the full-service unified data center of electric power system,a control scheduling model for the data processing task is given,and the priority of the task is analyzed synthetically from both the time and effective value of task.A multi-level queue based dynamic real-time priority scheduling（DRPS）algorithm is proposed,and the dynamically changing trend of priority for different types of tasks is analyzed.The experiment of data extract-transform-load（ETL）process of full-service unified data center is simulated on the CloudSim platform.The result shows that compared with the dynamic value-density（DVD）algorithm and the earliest deadline first（EDF）algorithm,the DRPS algorithm has a significant reduction in the deadline miss rate,and the completion rate of the task is significantly improved.