ZFS在基于HDFS的近线存储系统中性能分析与实现

Performance Analysis and Implementation of ZFS on Near-Line Storage Based on HDFS

通过将云计算技术应用于近线存储领域,设计实现了基于Hadoop分布式文件系统(Hadoop distributed file system,HDFS)的海量近线存储系统,用以扩展TH-1A超级计算机在线存储系统空间,提高系统利用率.该系统采用胖存储节点(大型的存储服务器)构建基于HDFS的近线存储集群,通过ZFS文件系统来解决HDFS存在的低效和空间利用率低的问题,采用双副本冗余、虚拟存储、动态管理等方法优化存储服务器存储和管理效率,提高了近线存储系统性能.实验结果表明,与传统HDFS集群存储系统相比,该系统在可靠性、经济性及灵活性上均具有一定的优势.

This paper designs and implements a near-line storage system based on Hadoop Distributed File System(HDFS),we apply the cloud computing technology into the field of near-line storage for the expansion of TH-1Asupercomputer online principal system storage space and to improve system utilization.The system uses a lot of large storage servers to structure near-line storage cluster based on HDFS,which takes the advantage of ZFS file system to solve the shortage of HDFS.By the use of double redundant copies,virtual storage,dynamic management for efficient storage and management of server storage media,we can improve system performance.Experimental results show that the system have certain advantages in terms of reliability,economy and elasticity compared to the traditional HDFS storage system.