基于云计算的海量大数据智能清洗系统设计

Design of intelligent cleaning system for massive data based on cloud computing

大数据资源是企业拥有的最重要的战略资源之一,也是管理层制定远景规划,提高市场竞争力的主要方式和途径,但大数据中会存在错误、冗余和不完整的数据,降低了大数据的总体质量。为此设计一种基于云计算的海量大数据智能清洗系统,改善现有大数据清洗系统在脏数据处理性能上的不足。分析了基于云计算的大数据智能清洗系统的总体框架和硬件构成,数据清洗系统的硬件部分由数据预处理模块、数据云聚类模块、数据识别模块和实体划分模块等部分组成;给出了智能大数据清洗系统的总体软件工作流程,并重点分析大数据空间聚类、相似度计算等关键的数据处理技术。验证结果表明,提出的基于云计算的海量大数据智能清洗系统设计的总体功能性较为完善,在系统性能测试方面也能够保持95%以上的数据查准率和召回率。

Big data resource is one of the most important strategic resources owned by the enterprise.It is also the main way and approach for the management to make long⁃term plans and improve market competitiveness of the enterprise.However,there are errors,redundancies and incomplete data in big data,which reduces the overall quality of big data.For this reason,an intelligent cleaning system for massive data based on cloud computing is designed to improve the shortcomings of the dirty data processing performance in the existing big data cleaning system.The overall framework and hardware structure of the intelligent cleaning system for big data based on cloud computing are analyzed.The hardware of the system consists of data preprocessing module,data cloud clustering module,data identification module and entity partition module.The overall software workflow of the intelligent cleaning system for big data is provided,and the key data processing technologies like big data clustering and similarity calculation are emphatically analyzed.The verification results show that the proposed intelligent cleaning system for massive data based on cloud computing has perfect overall functions and the precision and recall rate of data can be kept above 95%in the system performance testing.