一种基于发布/订阅机制的数据集成方法

Solution to data integration based on Pub/Sub system

数据集成包括数据的抽取、转换、加载三个过程。针对当前大多数据集成工具适应性差、源数据端和目的数据端的耦合度太紧和数据不易自动更新的问题,基于发布/订阅机制的消息代理模式提出了一种新型的数据集成体系结构,把数据集成划分为源数据处理(抽取转换)和目的数据处理(过滤加载)两个过程,不仅实现了数据的可靠异步传输,而且降低了源数据端和目的数据端的耦合度,增加了数据集成的灵活性,并使数据的抽取转换和过滤加载独立、并行执行。实验结果也证明了提出的数据集成方法有效地提高了数据集成的效率。

Data integration includes three phases：data extraction,data transformation and data loading.Poor adaptability,high coupling between source and destination and lacking of supporting automatic updating are main problems in most of data integration tools.In this essay,a new data integration architecture based on message Agent pattern of Pub/Sub mechanism technology is proposed,which divided the data integration into dealing of source （extraction and transformation） and destination （filtration and loading）,which not only realize reliable asynchronous transmitting,but also reduce coupling between source and destination,thus enhance the flexibility of data integration,and enable extracting,transforming and loading in a independent, parallel and efficient way.