面向金融行业的大数据迁移的研究与实现

Research and implementation of big data migration for financial industry

现有的金融行业的数据管理模式主要依赖于传统关系型数据库,然而传统架构受到拓展能力和存储性能的限制,难以满足大数据时代快速增长的海量数据量处理的需要。针对金融数据规模大、跨地域、跨系统存储、数据多样化等特点,提出了HiETL大数据迁移管理平台,实现了异构关系型数据库业务系统向Hadoop大数据平台的统一迁移,以及海量数据的集中整合、拓展存储、高效分析查询等一站式管理平台,在保证迁移准确的情况下,其速度可达到3 MB/s。

Current data management scheme in financial industry basically relies on traditional databases. However, due to the inherent limitation on the extension capability and storage performance, it is difficult to meet the rapid growing demands for massive data processing in big data era. Based on the characteristics of financial data, such as, large scale,diversity, cross region and cross storage system, this paper proposes a big data migration management platform, called HiETL, the functions of which include the data migration from heterogeneous relational databases to Hadoop platform,centralized integration of massive data, expanding storage, and efficient query. The migration speed can reach 3 MB/s as long as the migration accuracy is ensured.