基于频数的孤立点检测研究

Research on Frequency-based Outlier Mining

基于距离的孤立点检测算法在很多领域都有重要应用,效率不高却限制了孤立点检测算法的广泛应用。针对这个问题,文中通过分析基于索引的检测算法和基于单元的分析算法,受频繁项集挖掘算法的启发,应用统计学原理,提出了改进的基于距离的孤立点检测算法(Index Unit Based-on-Distance Outlier Mining,IU-BDOM)。在待挖掘数据集中,出现次数越少的项集越可能是孤立点,即频数越低越可能是孤立点,本算法在检测孤立点时,从频数最小的项开始检测,从而节省了挖掘频数很高的肯定不是孤立点的数据所带来的开销。为了进一步加快速度,实现算法的并行性,采用超立方体代替传统的超球体来统计数据集中每一个对象o的邻居个数,把距离的计算分散到不同的维度上独立进行,并且给予不同维度不同的权重,此外,利用Greenpulm分布式数据库,并行了挖掘任务,极大地提高了挖掘效率,并通过实验证实了这种改进的有效性。

Distance-based outlier detection algorithm in many fields has important applications, but the efficiency is not high which limit- ed the widely used outlier detection algorithms. For this problem,through analysis of the index detection algorithra and cell-based analysis algorithms,inspired by frequent itemsets mining algorithm, using statistical principles, proposed an improved distance-based outlier detec- tion algorithm （Index Unit Based-on-Distance Outlier Mining ,IU-BDOM）. Data to be excavated concentrated,appears more times the more less of the item sets may be an outliers, i. e. the frequency is the more low, the more likely is an outliers. The present algorithm in the detection of the outliers ,from the frequency of the minimum of the items start detection ,thereby saving the excavation frequency num- ber overhead high certainly not an outliers. In order to further accelerate the speed and realize the parallelism of the algorithm,the number of neighbors used the hypersphere to statistics hypercubes instead of the traditional centralized each object o, the distance independently calculated dispersed into different dimensions, and give different weights to different dimensions, in addition, the use of distributed data- base of Greenpulm, parallel mining tasks and greatly improve the efficiency of mining, and the effectiveness of such an improved is con- firmed by experiment.