面向海量数据存储的Erasure-Code分布式文件系统I/O优化方法

I/O optimization in Erasure-Code distributed file system for massive data storage

随着海量数据的快速膨胀,机群文件系统的存储方式正在逐步从复本向Erasure Code过渡。Erasure Code存储能够以更低的存储开销提供更高的可靠性。然而,由于Erasure Code存储需要通过编码生成编码数据,在存储原始数据和编码数据过程中更容易产生磁盘争用和不均衡负载,从而影响整个存储系统的I/O性能;同时,Erasure Code存储写回编码数据时,数据一致性和数据缓存之间存在冲突,传统处理数据的无缓存方式和全缓存方式在机群文件系统中都存在很大的局限性。针对这两个问题,提出了一种包括均衡负载的数据放置策略和编码缓存的一致性维护策略的Erasure Code机群文件系统I/O优化方法。通过在开发的Erasure Code分布式文件系统ECFS的实验测试表明,使用这种优化方法后机群文件系统的聚合带宽能够提高95.53%。

As the rapid growing of massive data, the storage method of cluster file system is developing from replication to Erasure Code. The storage system based on Erasure Code can provide higher reliability with less storage overhead. However, in the procedure of storing original data and the coded data, storage based on Erasure Code faces more disk I/O conflicts and unbalanced load, which jeopardizes the throughput of the system. Specially, there is a tradeoff between data consistency and data caching in the storage system based on Erasure Code when writing back the parity. And there are limitations for the use of non--data-caching machanism and all-data-caching machanism in the cluster file system. For these two issues, the paper proposed an I/O optimization method, which includes the data layout machanism balancing the load and the parity consistency machanism. In the cluster file system ECFS we developed base on Erasure Code, the throughput of the system can be improved by 95.53% after exploiting the I/O optimization.