一种基于日志结合分析的集群系统失效预测方法

A log co-analysis based failure prediction method for large-scale cluster systems

研究了大规模超级计算机群系统的失效预测。针对现有的单一分析系统日志的方法不仅需要复杂的分布式数据挖掘技术,而且失效预测的召回率普遍不高的问题,提出了一种通过将记载集群系统负载变化的作业日志同系统日志结合起来分析来进行失效预测的方法。该方法的原理如下:首先,通过对原始系统日志和作业日志进行预处理和过滤,分别得到细粒度的二维的事件序列和作业序列;然后从中抽取出在系统日志的失效事件发生前作业日志所表现出的三种典型失效征兆;最后,利用失效征兆进行失效预测。在真实的IBM Blue Gene/P系统的系统日志和作业日志上的实验结果表明,基于两种日志结合分析的方法能以较高的准确率和召回率实现细粒度的失效预测。

The failure prediction for large-scale cluster supercomputer was studied.Aiming at the prolem that the existing prediction method only analyzing the single system log needs complex data mining techniques while its prediction re-call rate is generally lower, this study presented an effective failure prediction method based on co-analysis of sys-tem logs and job logs that records the running workload information.The principle of the method is below：Firstly, the fine-grained two-dimensional event sequence and job sequence are produced through preprocessing and filtering of the two raw logs;Secondly, three failure symptoms are extracted from job logs before the occurrence of failure events;Finally, failure predictions are carried out by using these symptoms.The results of the experiments on real logs of the BlueGene/P system show that the proposed method can predict failures with a higher precision and a higher recall rate.