MUSE:一种面向云存储系统的高性能元数据存储引擎

MUSE: A High-Performance Metadata Storage Engine for Cloud Storage System

该文设计了一种高性能的面向云存储系统的元数据存储引擎(MUSE)。首先,其底层物理存储模块采用LSM-tree模型的高速key-value存储引擎Level DB方案,通过设计多缓存表和多线程紧凑机制对该方案进行优化,使其可以充分利用内存和多核CPU并行能力;其次,提出了基于多I/O通道的元数据存取调度机制。通道之间读写操作隔离,聚合多个通道为上层提供高并发随机I/O读写能力;此外,针对上层目录命名空间管理,提出路径分割映射和全路径映射策略两种策略,可基于不同的应用场景在性能与可用性间进行折中选择。系统测试结果表明,MUSE能够很好地适应海量小文件存储场景,相对于其他元数据存储系统在性能上有显著的提升。

In the cloud storage systems, the accesses of massive small files metadata will generate a large number of random disk I/O requests, which will become the performance bottleneck of the entire storage system. In this paper, metadata unit storage engine（MUSE）, a kind of metadata storage engine for cloud storage system, is proposed to support massive small files storage with high performance. Firstly, Level DB, a high speed key-value storage engine based on LSM-tree, is used as underlying physical storage module. Secondly, Level DB is enhanced by introducing multiple buffer tables and multiple compaction threads, which take full advantages of memory and multi-core processor. Thirdly, a new metadata accesses scheduling mechanism on multiple I/O channels is proposed. Channel is an independent data storage pipe formed by binding the independent thread to the independent physical disk. In this way, the access operations are isolated between channels, and then the aggregation of multiple channels can provide high concurrency random I/O. In addition, MUSE proposes two namespace management strategies： Split-path mapping strategy and absolute path mapping strategy, aimed to make trade-off according to different application scenarios by users. Benchmarks show that MUSE can support the massive small files storage scene and outperform other metadata storage systems.