摘要
主要从Map Reduce作业调度和Hive性能调优两个方面对Hive的性能优化进行研究.对于Map Reduce主要从编程模型切入,分析其执行过程,并从map端、reduce端进行参数调优.接着从Hive框架角度入手,分别从分区表和外部表以及常用数据文件的压缩、行式存储与列式存储等方面进行深入研究.实验结果表明,snappy压缩、orcfile/parquet存储格式对于列式查询,提高查询效率,对于大数据分析平台有较好的兼容性.
This paper research Hive performance optimization mainly from the two aspects of MapReduce scheduling and Hive performance tuning. MapReduce programming model and its implementation process is analyzed,and parameters are tuned from the map side and reduce side. Then Hive framework is researched from the aspects of the partition table,the external surface and common data file compression, the line storage and column type storage. The experimental results show that snappy compression and orcfile/parquet storage format can improve the efficiency of query for the column type queries, and has good compatibility.
出处
《上海师范大学学报（自然科学版）》
2017年第4期527-534,共8页
Journal of Shanghai Normal University(Natural Sciences)
关键词
数据仓库
作业调优
性能优化
压缩
存储格式
data warehouse
job optimization
performance optimization
compression
storage format
