一种面向数据可用性和存储可靠性动态要求的自适应纠删码存储策略设计

Design of an Adaptive Erasure Code Storage Strategy Oriented to Dynamic Requirements of Data Availability and Storage Reliability

为了满足指数级增长的大数据存储需求,现代的分布式存储系统需要提供大容量的存储空间以及快速的存储服务.因此在主流的分布式存储系统中,均应用了纠删码技术以节约数据中心的磁盘成本,保证数据的可靠性,并且满足应用程序和客户端的快速存储需求.在实际应用中数据往往重要程度并不相同,对数据可用性要求不一,且不同磁盘的故障率和可靠性动态不一的特点,对于传统RAID存储方式包括基于纠删码的存储系统提出了新的挑战.本文提出了一种面向数据可用性和磁盘可靠性动态要求的灵活自适应纠删码存储设计On-demand ARECS(On-demand Availability and Reliability Oriented Adaptive Erasure Coded Storage System),根据存储后端数据可用性和磁盘可靠性的多个维度进行设计,综合确定纠删码编码策略和存储节点选择,从而减少存储冗余度和存储延迟,同时提高数据可用性和存储可靠性.我们在Tahoe-LAFS开源分布式文件系统中进行了实验,实验结果验证了我们的理论分析,在保证具有多样性要求的数据可用性和磁盘可靠性的前提下,明显减少了数据冗余度和存储延迟.

In order to meet the exponentially growing demand for large data storage,modern distributed storage systems need to provide large-capacity storage space and fast storage services.Therefore,in mainstream distributed storage systems,erasure coding technology is applied to reduce disk costs in the data center,ensure data reliability,and meet the fast storage requirements of applications and clients.In practice,data often has different levels of importance and different requirements for data availability.Also,different disks have different failure rates and reliability.These pose new challenges for traditional RAID storage systems including erasure code-based storage systems.This article proposes On-demand ARECS(On-demand Availability and Reliability Oriented Adaptive Erasure Coded Storage System),which is designed according to multiple dimensions of storage back-end data availability and disk reliability,and comprehensively determines erasure coding strategy and storage node selection.This reduces storage redundancy and storage latency,while improving data availability and storage reliability.We conducted experiments in the Tahoe-LAFS open source distributed file system,and the experiment results verified our theoretical analysis.Under the premise of ensuring data availability and disk reliability with diverse requirements,data redundancy and storage delay were significantly reduced.