大数据环境下数据读取关键技术研究

Research on Data Reading Techniques Based on Big Data Environment

针对大数据环境下数据读取面临的主要挑战,文中重点研究了分布式文件系统中数据读取关键技术。根据数据存放结构的不同,从数据加载、查询处理和存储空间利用三个方面分析比较行存储、列存储和行列混合存储的优缺点和面临的挑战,重点介绍列存储中涉及到的压缩和物化技术,具体分析了存储压缩中经常运用的行程编码算法、词典编码算法、位向量编码算法和元组重构中运用的延迟物化技术。通过分析现有技术存在的问题,探讨相关的解决方案,并展望了未来研究的发展方向。

Under the big data environment, data reading has faced enormous challenges. In this paper ,focus on the key technologies of data in the distributed file system. Analyze the row-storage, column-storage, hybrid-storage according to data placement structure from data loading, query processing and storage space utilization. Besides, it introduces materialization techniques used in column-storage including run-length encoding, dictionary encoding ,bit-vector encoding and lazy decompression. Meanwhile, by analysis of the present problem, discuss the relative solutions, and has a prospect of future development.