数据中台架构的主数据平台及关键技术设计

Design of Master Data Platform and Key Technology of Data Center Architecture

针对传统数据平台存储容量不高、数据处理能力较差的问题,对业务数据进行层级分割和水平解耦,构建数据中台架构的主数据平台,实现跨域数据整合和数据积累。通过服务的形式构造数据接口,进行数据业务的开发,对业务前后端应用需求灵活应对。数据模块通过Instagram进行数据采集,数据注册模块使用Nginx+Gunicorn+Flask架构部署在AWS EC2服务器节点,多进程数据管理模式,快速处理多线程并发数据。实验结果表明,数据平台业务流程完整,数据传输没有误差,数据处理能力较强,运算效率较高。

Aiming at the problems of low storage capacity and poor data processing capabilities of traditional data platforms,this research completes hierarchical segmentation and horizontal decoupling of business data,builds a master data platform of data center architecture,realizes cross-domain data integration and data accumulation,constructs data interfaces in the form of services,and develops data services.It is flexible response to application requirements.The data module collects data through Instagram,and the data registration module uses the Nginx+Gunicorn+Flask architecture to deploy on AWS EC2 server nodes.Multi-process data management mode can quickly process multi-threaded concurrent data.The experimental results show that the data platform has complete business process,no error in data transmission,strong data processing capabilities,and high computational efficiency.