基于大数据的元模型及数据保障研究

Research on Meta Model and Data Guarantee Based on Big Data

针对海量的数据资源管理效率与质量问题,提出一种基于Hadoop的元数据管理模型。在该模型中,采用Hadoop对元模型进行搭建,从而完成对整个元数据的储存;然后通过ETL工具,通过分布式抽取的方式,实现对不同服务集群中化工类数据库数据的抽取,并通过ETL抽取、转换、加载和映射,建立映射规则,进而将不同的数据存储到对应的数据库中。最后,以某化工类数据库中的元数据抽取为例,通过对比数据集成率和数据可用性,看出本研究方案可提升海量数据下的数据有用率和集成率,对保障元数据的抽取质量具有借鉴价值。

Aiming at the efficiency and quality of massive data resource management,a metadata management model based on Hadoop is proposed.In this model,Hadoop is used to build the meta model,so as to complete the storage of the whole metadata.Then,through ETL tools,through the way of distributed extraction,the extraction of chemical database data in different service clusters is achieved,and through ETL extraction,transformation,load⁃ing and mapping,we can establish mapping rules,and then store different data in the corresponding database.Fi⁃nally,taking the metadata extraction in a chemical database as an example,by comparing the data integration rate and data availability,we can see that this research scheme can improve the data availability and integration rate un⁃der the massive data,which is of reference value to ensure the quality of metadata extraction.