基于CDC机制的数据仓库实时数据更新方法研究

Study on Method of Data Warehouse Real-time Data Updating Based on Mechanism of CDC

分析了某特定应用系统的数据仓库实时决策需求,确定了需要实时更新到数据仓库的数据库表。对几种实时更新数据的方案进行了比较和权衡,经过综合考量,设计了使用基于读取和分析数据库日志的CDC机制来捕获变更数据,然后在数据加载程序中设定周期,循环地将捕获到的变更数据放入中间数据集并批量加载到数据仓库中的实时数据更新方案。该方案经过实验验证可以满足实时更新数据的需求,并且更新数据的过程不会影响源系统的事务处理,适用于此应用系统。

This article analysed the real-time decision requirement of the data warehouse of a specific application system,identified database tables which need to be updated in real-time to data warehouse.Then compared several overall plan of real time data loading,after comprehensive consideration,we designed a method of real time data loading—using the CDC mechanism to acquire the changed data,and then circularly load the changed data to data warehouse via data loading program.After experimental verification,this method can meet the requirement of real time data loading,while doesn't have much influence over the Database Transaction.So it's suitable to be applied to the system.