一种新颖的基于密度的祛噪声方法

A Novel Algorithm for Outlier Removal Based on Density

由于采集技术和设备的局限,以及外界的各种干扰,采集得到的数据中常常参杂着噪声,直接影响着后续数据分析的结果.传统的祛噪声方法,或是依赖于数据服从某一特定分布的假设,或是只能对服从单一分布的数据进行祛噪声处理,这些固有的缺陷大大降低了处理后数据的可信度.本文提出了一种新颖的基于密度的祛噪声方法,并应用在实际交通数据的处理中.通过与传统方法的实验比较,结果表明该方法摒除了传统方法的缺陷,能够对服从多个相异分布的数据进行有效的祛噪声处理,且处理后的数据能够很好地保留系统本质的特征.

Due to the limitation of the present techniques and facilities for data collection and various interferences, the data obtained are often distorted and noised, directly influencing the result of subsequent data analysis. The conventional approaches to outlier removal either assume that the data follow a certain known distribution or deal with the data that are from a single distribution, resulting in a reduced credibility of the data processed. This paper proposes a novel method to remove outliers based on density estimation and it has been applied to real-world traffic data. By comparison with the conventional approach, the experimental results indicate that the proposed algorithm is capable of detecting and removing outliers effectively for the data that may follow different unknown distributions, and the processed data retain the original and significant characteristics possessed by the system.