基于开源代码的ETL工具的实现

The realization of ETL tool based on open-source

在企业数据愈加庞大的今天,大批量数据交换也越加频繁,ETL是数据处理的主要技术手段,但极少有ETL工具能够兼顾自动调度及跨网段交换。在开源ETL工具Kettle的基础上,设计实现能够定制触发规则、根据调度周期自主运行交换作业的数据交换工具GoData,该工具支持跨节点跨网段的数据交换作业,同时具有节点监控及错误作业预警和消息推送功能,支持集群和分布式部署。该工具极大地满足了企业对数据交换业务的需求,在企业合理和充分利用现有数据资源方面发挥作用。

Today, the enterprise data increases rapidly, and large quantities of data exchange also becomes frequent. The ETL is the main tech- nical means in this area, hut very few of the them can both automaticly schedule and exchange across inter network. Based on an open-source ETL tool named Kettle, we design the data exdange tool called GoData, which can set the trigger rules, and can run the ETL jobs on time all by itself according to the quartz period the tool. This tool can also support the inter network data exchange. At the same time, the tool has the functions of node monitoring and operation warning. It also can push the error message to client. This tool can support cluster and distributed deployment. It greatly satisfy the demand for data exchange business, and plays a role in making full use of the existing data resources.