分布式数据集成管理工具的研究与实践

Research and Practice of Distributed Data Integration Management Tools

随着大数据时代的到来,企业管理着规模庞大的数据资产,为满足企业日常批量数据集成、数据批量共享交换、数据分析的需求,本研究针对企业日常批量数据集成、数据批量共享交换、数据分析的工作的现状和难点,以及传统集中式数据集成管理工具存在的不足,提出了分布式数据集成管理工具解决方案,通过多节点、多线程的作业模式和智能化作业调度关键技术研究,完成了分布式数据集成管理工具搭建,并进行了试点应用,完成50亿条数据的抽取转换工作从82小时缩短至16小时,12亿条数据的抽取转换工作从19小时的抽取工作缩短至4小时,有效提升了数据集成效率。

With the arrival of large data age, enterprises manage large-scale data assets. In order to meet the needs of daily batch data integration, data batch sharing and data analysis, this study aims at the daily batch data integration, data batch sharing exchange,The status quo and difficulties of data analysis, and the shortcomings of traditional centralized data integration management tools, and proposed a distributed data integration management tool solution, through multi-node, multi-threaded mode of operation and intelligent operation scheduling key technology research, Completed the distributed data integration management tools to build, and conducted a pilot application to complete the extraction of 5 billion data conversion work from 82 hours to 16 hours, 1.2 billion data extraction conversion work from 19 hours of extraction work shortened to 4 hours, effectively improving the data integration efficiency.