Hadoop日志压缩算法的研究与实现

Research and Implementation of Hadoop Log Compression Algorithm

Hadoop集群由于规模增大、运行时间增长使得日志量持续增加,导致存储压力过大。为了节约存储资源,研究并实现一种高效的Hadoop日志无损压缩算法。实验结果表明,该压缩算法与传统的LZW压缩算法的压缩比基本相同,但压缩和解压操作的耗时均小于LZW压缩算法且受文件大小影响更小。

The increased size and run-time growth of Hadoop cluster make the log data continued to increase, which leads to the large storage pressure. In order to save storage resources, studies an efficiently lossless compression algorithm of Hadoop log. The results show that the compression algorithm proposed is basically the same as the compression ratio of LZW compression algorithm, but the compression, decompression operation time of the proposed algorithm is much shorter than the LZW compression algorithm, and the proposed algorithm is less affected by the file size.