Hadoop集群异常节点实时检测与诊断算法

Hadoop anomaly real-time detection and diagnosis algorithm

针对Hadoop集群节点增加导致任务运行效率降低,以及异常节点会拖慢整体作业进度的问题,提出了一种Hadoop集群异常节点实时检测与诊断算法。首先基于正常状态下节点性能相似性原理,使用Logstash工具收集Hadoop集群节点运行日志中的任务状态信息;其次,发现异常节点后,通过Perf性能分析工具收集体系结构性能信息,再利用异常节点诊断算法诊断导致该节点异常的原因。通过实时流计算框架Spark Streaming构建了异常节点实时检测与诊断模型,并设计了一系列的实验验证了本算法的有效性。

The problem that the increase of Hadoop cluster nodes leads to the decrease of task efficiency and the delay of the overall job schedule by abnormal nodes.This paper presents a real-time detection and diagnosis algorithm for outliers in Hadoop cluster.Firstly,based on the principle of node performance similarity under normal state,the task status information in Hadoop cluster node running log is collected by using the Logstash tool.Secondly,after discovering the abnormal node,the architecture performance information collected by the Perf performance analysis tool is used to diagnose the cause of the abnormal node by using the abnormal node diagnosis algorithm.A real-time detection and diagnosis model of abnormal nodes is built by spark streaming,and a series of experiments are designed to verify the effectiveness of the algorithm.