Hadoop小文件编程处理的性能优化

Small File Performance Optimization Programming Processing Based on Hadoop

针对Hadoop在处理海量小文件时,存在的性能问题进行了源码分析。研究了默认的输入格式TextInputFormat导致性能低下的原因,阐释使用CombineFileInputFormat作为输入格式的编程方法及其对性能提升的原因。以垃圾邮件训练集分析为例,介绍了海量小文件编程处理性能优化的具体实现方法。实验表明CombineFileInputFormat输入格式对计算性能的提升起到了显著的作用。

This paper analyzes the problem from code analysis,studies the poor performance reason that use default input format-TextInputFormat,explains the programing method and the performace improvement reason that use CombineFileInputFormat as input format.Then the paper uses training set analysis of spam mail as example,to introduces how to optimize programming processing performance for massive amounts of small file.