基于大数据分析的海量数据离群点检测方法

Outlier Detection Method For Massive Data Based on Big Dat a Analysis

由于传统的海量数据离群点检测方法,在进行海量数据离群点检测时存在计算不准确、检测精度低等问题,笔者提出基于大数据分析的海量数据离群点检测方法:建立R2-tree大数据结构离群点检测树,不断地调整大数据结构离群点数目;计算海量数据离群点检测阈值;寻找与其他数据样本一般行为不同或特征差异性大的点,完成海量数据离群点检测。实验结果表明,在对相同个数的数据进行检测时,设计方法正确检测到海量数据离群点的个数高于传统方法,能够实现对海量数据离群点的精准检测。

Because of the traditional outlier detection method of mass data,there are some problems in the outlier detection of mass data,such as inaccurate calculation and low detection accuracy.The author proposes the outlier detection method of mass data based on big data analysis:establishing R2 tree outlier detection tree of big data structure,constantly adjusting the number of outliers of big data structure;calculating outlier detection threshold of mass data Find out the points with different general behaviors or features from other data samples,and complete the outlier detection of massive data.The experimental results show that when the same number of data is detected,the number of outliers detected by the design method is higher than that of the traditional method,which can achieve the accurate detection of outliers.