一种面向Hadoop中间数据存储的混合存储系统

Hybrid storage system focus on Hadoop intermediate storing

针对Hadoop的Reduce阶段的I/O性能瓶颈,设计了针对Hadoop中间数据存储的设备同层混合存储系统。该系统包含数据映射和数据迁移两大模块,其中数据映射模块主要负责I/O请求重定向及维护元数据,数据迁移模块利用闲时进行数据迁移,提高SSD的利用率。实验结果表明,该混合存储系统能够缓解Hadoop的Reduce阶段的I/O瓶颈并提升该阶段的整体性能。

For improving the performance of the Reduce phase of Hadoop caused by the I/O bottleneck,a new flat leveling device mapping hybrid store system has been designed focused on Hadoop intermediate data storing. The hybrid system consists of two part. Data mapping module＇s duty is to redirect the I/O request and maintain the metadata of hybrid store. Data migration module will migrate data during nonbusy time for improve the usability of SSD. The experimental results demonstrate that this hybrid store system can improve the Reduce phase＇s performance of Hadoop by remission the bottleneck of reduce phase.