科技创新大数据清洗框架研究

Research on big data cleaning framework of scientific and technological innovation

河北省科技创新大数据公共平台是以海量数据资源为基础,基于数据仓库与数据挖掘技术构建,面向管理部门开展决策服务,面向社会公众开展信息服务的网络信息平台。但是,在构建数据仓库的过程中,存在各种各样的数据质量问题,最终产生各种错误的分析结果,所以,在进入数据仓库前,必须对数据进行清洗,从而保证进入数据仓库的数据质量。根据河北省科技攻关项目科技大数据标准化处理与应用系统,提出科技创新大数据清洗框架,在框架的基础上,定义清洗规则,改进清洗算法,在真实数据集上进行实验,解决了进入数据仓库的数据质量问题,从而保证了数据在数据仓库中的一致性和正确性,为后期的分析和处理提供了坚实的数据基础。

Hebei Province science and technology innovation big data public platform is based on massive data resources,the construction is based on data warehouse and data mining technology,oriented management departments to carry out the decision-making service,network information platform for the public to provide information service.However,during the construction of data warehouse,there are all kinds of data quality problems,resulting in various error analysis results,so,before the data get into the data warehouse,data cleaning should be done,so as to ensure the quality of the data into data warehouse.According to the scientific and technological big data standardization processing and application system of science and technology project in Hebei Province,put forward the innovation of science and technology of data cleaning framework,on the basis of the framework,the definition of data cleaning rules,improved data cleaning algorithm,experiments were carried out on the technological innovation of large data system on real data sets,solving the problems of data quality in data warehouse,so as to ensure consistency and the correctness of the datain the data warehouse,providing a solid foundation for data analysis and processing of the late.