一种基于Agent和活动优先度的ETL过程并行方法

A parallel method for ETL process based on agent and activity priority

ETL是数据仓库获得高质量数据的关键环节,在数据仓库的构建和实施中占有重要地位。针对传统ETL串行执行方式的不足,提出一种基于Agent和活动优先度相结合的ETL并行执行方法。该方法计算ETL执行过程中各个活动的优先度,利用Agent理论和多线程并行计算技术实现并行执行具有相同优先度且相互间没有依赖关系的ETL活动。实验结果表明,该方法在数据量较大时具有较好的加速比,提高了ETL过程的执行效率。

ETL is the essential step to obtain high-quality data for data warehouse, and plays an important role in the construction and implementation of data warehouse. Aiming at the deficiency of traditional serial ETL process, we propose a parallel method for ETL based on agent and activity priority. This method first calculates the priority of each ETL activity and then utilizes the agent theory and multi-thread computing techniques to achieve parallel execution of independent ETL activities with the same priority. Experimental results show that this method achieves high speedup when the data volume is large and improves the efficiency of ETL process.