基于Hadoop的动态网络异常节点智能检测方法

Intelligent Detection Method of Abnormal Nodes in Dynamic Network Based on Hadoop

针对网络流量的急剧增加,导致网络集群中存在异常节点,降低了网络的运行效率等问题,提出基于Hadoop的动态网络异常节点智能检测方法。对Hadoop的生态体系结构进行分析,并详细描述了HDFS、MapReduce和NetBASS框架。采用Zipf排名分布对网络节点总流量进行统计,通过标准差系数对节点参数的稳定性和离异程度进行评价。为了使聚类效果更加稳定,使用新的均衡化函数作为数据的收敛函数。将异常节点检测分为节点检测和异常点定位两部分进行,诊断系统通过对Hadoop运行日志的实时检测来计算逻辑,系统通过作业中节点的表现情况,结合性能指标分析出该节点异常的原因,同时将map与reduce任务一起考虑的方法降低节点的耦合性。实验结果表明,通过在运行负载前和运行负载时故障的引入测试,验证了所提方法不仅能够准确的检测出异常点,而且有效性得到显著提高。

Due to the rapid increase of network traffic,there are abnormal nodes in the network cluster,which reduces the operation efficiency of the network.Therefore,an intelligent detection method of dynamic network abnormal nodes based on Hadoop is proposed.The ecological architecture of Hadoop is analyzed,and the HDFS,MapReduce and NetBASS frameworks are described in detail.Zipf ranking distribution is used to count the total traffic of network nodes,and the stability and divorce degree of node parameters are evaluated by the standard deviation coefficient.In order to make the clustering effect more stable,a new equalization function is used as the data convergence function.Finally,abnormal node detection is divided into two parts:node detection and abnormal point location.The diagnosis system calculates the logic through the real-time detection of Hadoop operation logs.The system analyzes the cause of the node abnormality through the performance of the node in the job and combined with the performance index.At the same time,the method of considering map and reduce tasks together reduces the coupling of the node.The experimental results show that the proposed method can not only accurately detect abnormal points,but also significantly improve its effectiveness.