基于大数据技术的数据仓库体系结构设计

Research on Data Warehouse Architecture Based on Big Data Technology

大数据时代,数据结构的复杂性和数据体量的剧增使得传统数据仓库体系结构已经不能满足数据处理的需要。本文在对传统数据仓库体系结构相关理论研究的基础上,重点分析了传统数据仓库体系结构存在的不足以及大数据的特征和对数据处理的新需求,构建了基于大数据技术的数据仓库体系结构,采用目前流行的Hadoop/Spark大数据处理技术架构,实现半结构化数据和非结构化数据的收集、处理、存储和分析挖掘,弥补了传统数据仓库在海量数据处理、存储方面的不足,有效解决数据利用率低、价值发挥不明显的问题。

In the age of big data,the complexity of data structure and the sharp increase of data volume make the traditional data warehouse architecture connot meet the needs of data processing.This paper is based on the theoretical research of traditional data warehouse architecture, shortcomings of traditional data warehouse architecture and characteristics of data and new requirements for data process are emphatically analyzed,the system architecture of data warehouse based on big data technology is constructed,adopt the currently popular hadoop/spark big data process technology architecture,realizes the collection,processing,storage ,analysis and mining of semi-structured and unstructured data , makes up for the deficiency of traditional data warehouse in mass data processing and storage,this way can solve the problem of low utilization of data and unobvious value.