CarrierFS:基于虚拟内存的分布式文件系统

CarrierFS:a virtual memory based distributed file system

针对由于磁盘I/O的性能瓶颈,基于磁盘的分布式文件系统无法为海量小文件存储提供高数据吞吐率和低访问延迟这一问题,设计并实现了一个基于虚拟内存的分布式文件系统——CarrierFS,它将数据存储于虚拟内存中,利用磁盘备份数据.在实现元数据和数据可扩展存储的基础上,CarrierFS设计了监管者模块来保证系统的可靠性.对CarrierFS和HDFS进行了性能测试,实验结果表明,CarrierFS对大文件和小文件的读写速度都具有明显的性能优势.当文件大小为256 Mbyte时,CarrierFS的写入速度大约是HDFS的1.30倍,读取速度大约是HDFS的1.95倍.当文件大小为32 Kbyte时,CarrierFS的写入速度大约是HDFS的3.0倍,读取速度大约是HDFS的3.4倍.

Existing disk-based distributed file systems suffer from performance degradation on small files,as a result of the I/O bottleneck of disk.This paper demonstrates that significant performance gains can be achieved if,instead,all the data can be stored in memory.We explore the design and implementation of such a memory-based distributed file system,called CarrierFS, which also ensures the reliability of system by introducing supervisor modules and offers reliable storage by using disk for backup.Experimental results show that CarrierFS provides a high performance solution both on small and large files.Compared to HDFS(Hadoop Distributed File System),it increases both read and write performance on small files by a factor of 3.0～3.4,and on large files by a factor of 1.30～1.95.