一种基于Kettle的无损增量数据同步方法研究

A Lossless Incremental Data Synchronization Method Based on Kettle

在学校各系统运行管理过程中,产生了大量宝贵的数据资源,这些数据对学校重要决策制定及传统教育模式改革有着极其重要的作用。因此,如何将不同维度的数据采集到统一的数据中心便成为大数据研究的重点之一。在数据采集过程中,很多学校没有保存重要的历史数据以及已删除记录的状态标记,将对数据分析中诸如时间切片分析、历史状态分析等产生致命影响。以学校人事系统为例,提出一种基于Kettle的无损增量数据同步方法。该方法利用全量数据比对方式,找出新增、修改和删除的数据,并对其进行详细记录,从而实现了对历史数据的完整保留,弥补了如时间切片分析等数据分析策略中数据不足的缺陷。

The operation and management of school system generates a large number of valuable data resources.These data play an extremely important role in making important school decisions and reforming the traditional education model.How to collect these different dimensions of data into the data center has become one of the focuses in big data research.In the process of data acquisition,most schools do not pay attention to the preservation of historical data and the historical trace records of deleted data,which will have a fatal impact on data analysis such as time slice analysis,historical state analysis and so on.Taking the school personnel system as an example,this paper proposes a Kettle-based lossless incremental data synchronization method.This method uses the way of full data comparison to find new,modified and deleted data,and record them in detail.It realizes the complete preservation of historical data and fills in the shortcomings of data analysis strategies such as time slice analysis.