面向回填优化的作业时长预测

Runtime Prediction of Jobs for Backfilling Optimization

高性能计算集群通常使用先来先服务等传统的作业调度方法,它具有良好的公平性,并且实现简单,但容易产生闲置的资源碎片.针对上述问题,一种的策略是使用回填,利用一些短时间小作业来填补系统等待期间的空闲资源碎片.但好的回填方法通常需要知道作业预期的运行时间,而用户或者不愿意提供作业预期运行时间,或者倾向于提供比实际运行时间更长的预期时间以避免作业被系统终止,因此我们有必要自行预测作业的运行时间. VASP是国内应用最普及的高性能计算应用软件之一,本文通过分析VASP作业特性,解析并抽取相应的作业特征集,提出一种基于贝叶斯的二次预测模型IRPA,对VASP作业进行运行时长的预测,最后进一步提出基于径向基网络分支及贝叶斯分类的混合预测模型BRBF,并且利用我校TC4600平台上的VASP作业数据集进行验证.实验结果和其他几个基本方法进行对比,表明IRPA以及BRBF的有效性以及在粗粒度下具有的较高预测准确率.

The most common scheduling strategy used on high performance computing systems is First Come First Server( FCFS). The FCFS strategy has good fairness and is simple and practical,but it may generate idle resource fragments. The backfilling algorithm is one of the wildly used method to improve the utility of the system. It let some short runtime small jobs run ahead to fill the blank resources which are waiting for large jobs. The backfilling algorithm depend on the knowledge of jobs runtime before they really run.However,users may not willing to provide their jobs runtime or the provided runtime are much longer than actually runtime to avoid their jobs be terminated by the system. Therefore,it is necessary to predict the jobs runtime based on the jobs properties. The Vienna Ab initio Simulation Package( VASP) is one of the most popular high performance computing applications. We extracted the corresponding job attribute sets by analyzing the characteristics of VASP,and presented a job runtime prediction method based on Bayesian model: IRPA,which is used to predict the running time of VASP jobs. We also proposed a hybrid method based on radial basis network and Bayesian model: BRBF. These two models are verified the two models by using the history data of VASP jobs on TC4600 platform in supercomputer center of University of Science and Technology of China( USTC). Compared to some other classical methods,our two methods showthe better effectiveness and higher prediction accuracy at a coarse granularity.