摘要
针对水利普查数据海量、多维的特点,研究近年来在"大数据"概念下发展迅速的Hadoop与Hive,结合传统数据仓库在多维数据分析方面的成熟技术,提出基于Hive的水利普查数据仓库的构建方法,描述数据仓库系统的架构,并根据Hive的设计特点,通过分桶、消减维度表和冗余事实表的方法来改进传统的多维分析模型,最后搭建集群系统对水利普查数据集进行查询与分析测试。测试结果表明该数据仓库可以满足海量多维水利普查数据的存储与查询要求。
For the characters that water census data is of large volumes and high dimension, studying Hadoop and Hive which have a quick development recently in the ＂big data＂ concept and combining mature technology in multidimensional data analysis using traditional data warehouse, this article proposes a construction method of water census data warehouse based on Hive. This paper describes the architecture of data warehouse system, improves multidimensional model by dimension table reduction, fact table redundancy and Hive＇ s bucket method, then carries on queries and analysis to water census data set on Hadoop cluster system. Experimental results show that the data warehouse meets the f storage and query requirements of massive multidimensional water census data.
出处
《计算机与现代化》
2014年第5期127-130,共4页
Computer and Modernization
基金
国家自然科学基金资助项目(51079040)
水利部948项目(201016)
关键词
数据仓库
水利普查
模型优化
大规模数据处理
Hive
data warehouse
water census
model optimization
large data processing
