节点实时性能自适应的集群资源分配算法

Node real-time performance adaptive cluster resource scheduling algorithm

由于配置和所运行作业的不同,集群各节点的实时性能差异较大。为提高集群性能,提出节点实时性能自适应的集群资源分配算法(node real-time performance adaptivecluster resource scheduling algorithm, NPARSA)。节点实时性能用其配置(CPU核数及速度、内存容量、磁盘容量)和实时状态参数(CPU、内存和磁盘的剩余数量及磁盘读写速度)表示。NPARSA根据作业类型自主选择节点性能评价指标的权值,实现节点实时性能对于作业类型的自适应。实时性能最优的节点分配给作业。虚拟机实验和物理集群实验表明,与Spark默认资源分配算法、没有考虑作业类型与节点匹配的算法、使用作业和节点匹配差异程度作为资源分配依据的算法相比,NPARSA能更有效地缩短作业执行时间、提高集群性能。

The real-time performance of each node in one cluster varies greatly due to different configurations and the job running on it. To improve the cluster performance, NPARSA(node real-time performance adaptive cluster resource scheduling algorithm) was proposed. The real-time performance of a cluster node was represented by its configuration(such as the number of its CPU cores, the speed of CPU, memory capacity, and disk capacity) and the real-time state parameters(such as the residual of CPU, memory, and disk). NPARSA chose the attribute weights for a node according to the type of the job to be handled, and assigned nodes with higher priority to the job. Virtual machine experiments and physical cluster experiments prove the effectiveness of NPARSA. Compared with Spark′s default scheduling algorithm, the algorithm that does not consider the job type and node matching, and the algorithm that uses the degree of job and node matching difference as the basis for resource allocation, NPARSA can improve the performance of a cluster and shorten the execution time of user jobs.