企业轻量化大数据架构研究

Research on the Big Data Architecture of the Lightweight of Enterprises

对于很多中小型的大数据项目,应用MySQL等关系型数据库无法实现大数据的存储与计算,应用传统Hadoop大数据生态又太重,比较耗费人力、财力、服务器等资源。该文研究企业轻量化大数据架构的一种落地方案,并分析其应用场景。该文研究的轻量化大数据架主要针对企业应用中绝大多数结构化和半结构化大数据分析,数据量规模在1TB到10PB之间。轻量化架构采用MPP数据库(Doris)作为底层存储和计算引擎,Kafka作为数据接入缓冲通道,开发一体化轻量管理组件实现大数据开发中常用的任务调度、表管理、SQL开发、数据接入等功能。

For many small-and medium-sized big data projects,the application of relational databases such as MySQL cannot realize the storage and computing of big data,and the application of traditional Hadoop big data ecology is too heavy and consumes more resources such as manpower,financial resources and servers.This paper studies a landing scheme of the big data architecture of the lightweight of enterprises and analyzes its application scenarios.The big data architecture of lightweight studied in this paper is mainly for the analysis of most structured and semi-structured big data in enterprise applications,and the data size is 1 TB~10PB.The architecture of lightweight uses the MPP database(Doris)as the underlying storage and computing engine,and uses Kafka as the buffer channel for data access,and develops integrated lightweight management components to achieve the commonly-used functions such as task scheduling,table management,SQL development and data access in big data development.