摘要
针对由于磁盘I/O的性能瓶颈,基于磁盘的分布式文件系统无法为海量小文件存储提供高数据吞吐率和低访问延迟这一问题,设计并实现了一个基于虚拟内存的分布式文件系统——CarrierFS,它将数据存储于虚拟内存中,利用磁盘备份数据.在实现元数据和数据可扩展存储的基础上,CarrierFS设计了监管者模块来保证系统的可靠性.对CarrierFS和HDFS进行了性能测试,实验结果表明,CarrierFS对大文件和小文件的读写速度都具有明显的性能优势.当文件大小为256 Mbyte时,CarrierFS的写入速度大约是HDFS的1.30倍,读取速度大约是HDFS的1.95倍.当文件大小为32 Kbyte时,CarrierFS的写入速度大约是HDFS的3.0倍,读取速度大约是HDFS的3.4倍.
Existing disk-based distributed file systems suffer from performance degradation on small files,as a result of the I/O bottleneck of disk.This paper demonstrates that significant performance gains can be achieved if,instead,all the data can be stored in memory.We explore the design and implementation of such a memory-based distributed file system,called CarrierFS, which also ensures the reliability of system by introducing supervisor modules and offers reliable storage by using disk for backup.Experimental results show that CarrierFS provides a high performance solution both on small and large files.Compared to HDFS(Hadoop Distributed File System),it increases both read and write performance on small files by a factor of 3.0~3.4,and on large files by a factor of 1.30~1.95.
出处
《华中科技大学学报(自然科学版)》
EI
CAS
CSCD
北大核心
2010年第S1期37-42,共6页
Journal of Huazhong University of Science and Technology(Natural Science Edition)
基金
国家自然科学基金资助项目(60803121
60773145
20091351008
90812001
60963005)
国家高技术研究发展计划资助项目(2006AA01A101
2006AA01A108
2006AA01A111
2009AA01A130)
国家科技支撑计划资助项目(2006BAK15B10)
关键词
分布式文件系统
存储系统
虚拟内存
小文件
吞吐率
访问延迟
distributed file system
storage system
virtual memory
small file
throughput
access latency