数据预处理技术在异构数据中的应用

Application of Data Preprocessing Technology in Heterogeneous Data

数据预处理成为数据挖掘与分析最重要的一环,针对多源数据中数据冗余和数据缺失以及浪费算力的问题,本文采用系数对重复性的数值型数据进行检测和基于传统的字段匹配算法,提出了系数能够解决相同的字段在不同的字符串中的相似程度的问题,对缺失值数据采用频率分布和牛顿插值法来填充缺失值,使数据也具有基本的安全性和有效性,采用马氏距离可以有效的去除异常值,最后通过数据约简的方法来减少数据规模。经实验验证,经过预处理后的数据建立的模型的准确度没有太大影响,但处理时间以及安全性得到了大幅度的提升。

Data preprocessing has become the most important part of data mining and analysis.Aiming at the problems of data redundancy and data loss and wasted computing power in multi-source data,this paper uses Jaccard coefficient to detect repetitive numerical data and based on traditional Field matching algorithm,proposed the R-ppd coefficient can solve the problem of the similarity of the same field in different strings,using Laplace distribution and Newton interpolation for missing data to fill the missing values,making the data also It has basic security and effectiveness.Using Markov distance can effectively remove outliers.Finally,data reduction method is used to reduce the data size.The experimental verification shows that the accuracy of the model established by the preprocessed data does not have much impact,but the processing time and safety have been greatly improved.