基于Hadoop搭建高可用数据仓库的研究和实现

Research and Implementation of Building High Availability Data Warehouse Based on Hadoop

目前,网购交易的日益增加使得电商数据量呈现疯狂增长的态势,数据量的大增需要引入数据仓库,用以支持对大容量数据的分析和处理。在数据仓库的架构设计过程中,将HDFS作为底层文件存储系统,避免因某些原因而导致的系统崩溃。该文对高可用数据仓库的应用进行深入的研究,通过搭建高可用数据数仓平台,解决Hadoop单节点故障问题,提高数据采集和存储的效率,有效解决了传统数据分析的局限性,具有一定的应用推广价值。

At present,the growing number of online shopping transactions has led to a crazy growth of E-commerce data volume,which requires the introduction of data warehouses to support the analysis and processing of large volume data.During the architecture design of the data warehouse,HDFS is used as the underlying file storage system to avoid system crash for some reasons.This paper conducts in-depth research on the application of high availability data warehouse.By building a high availability data warehouse platform,it solves the problem of Hadoop single node failure,improves the efficiency of data collection and storage,effectively solves the limitations of traditional data analysis,and has certain application promotion value.