浅析基于ETL技术和全表扫描哈希对比的数据同步方法

Ilustration of Data Synchronization Method Based on ETL Technique and a Full Table Scan with Hash Comparison Algorithm

数据同步技术随着企业对各个信息系统之间数据共享的重视而变得越来越重要,数据同步的方法有很多,本文主要介绍基于ETL技术和基于全表扫描及哈希对比两种数据同步方法。基于ETL技术的数据同步是基于中间逻辑表完成数据转换,再通过主键和时间戳的对比而完成数据同步过程,本文以Kettle工具为例进行分析;基于全表扫描及哈希对比的数据同步是基于视图完成数据转换,再通过hash算法扫描对比而完成数据同步过程。

With more and more attention being paid to data sharing among internal information systems of a corporate, data synchronization method is becoming more and more important. Among many ways of data synchroniza- tion, two will be introduced in this paper： ETL and full table scan with hash comparison algorithm. ETI -based technique completes data synchronization through data transformation on the basis of logical table and then the com- parison between primary key and timestamp data. Taking Kettle tool as an example, this paper will further illustrate the approach by transformation of data on view, using full table scan with hash comparison algorithm, followed by scan comparison with hash algorithm.