键值对与对象存储在分布式存储系统中的应用

Application of K-V pair and object storage in distributed storage system

为解决传统存储系统在海量文件存储和文件路径快速变更时难以兼顾的问题,在对比分析文件存储和对象存储利弊的基础上,提出将K-V键值对的文件目录结构与对象存储结合的方案,使系统在拥有高吞吐量的同时,快速完成目录的迁移工作。在使用TiKV集群存储文件元数据提高数据扩容上限时,引入虚拟节点的哈希算法与代理节点结合,减少集群扩容时的IO漂移。实验结果表明,该系统具有吞吐量大、迁移速度快、水平扩展性强、扩容过程平稳的特点。

To solve the problem that the traditional storage systems fail to consider storing massive files and rapidly changing file paths at the same time,based on the comparative analysis of the advantages and disadvantages of file storage and object storage,a scheme combining the file directory structure of K-V key value pairs with object storage was proposed,so that the system could quickly complete the directory migration while having high throughput.When using the TiKV cluster to store file metadata to improve the upper limit of data expansion,the hash algorithm of virtual node was combined with proxy node to reduce the IO drift during cluster expansion.Experimental results show that the system has the characteristics of large throughput,high migration speed,strong horizontal scalability and stable capacity expansion process.