分布式文件系统中海量小文件异步创建技术

An Asynchronous Creation Technology for Massive Small Files on Distributed File System

海量小文件应用已经渗透到高性能计算、企业办公、互联网等多个领域。由于小文件主要存储在分布式文件系统中,海量小文件访问性能已经成为分布式文件系统亟待解决的难题。影响性能的关键因素是高比例高并发的元数据同步访问,尤其在高并发小粒度创建应用中,元数据同步网络交互增加了小文件访问延迟。本文以p NFS分布式文件系统为基础,研究降低海量小文件创建应用延迟的关键技术,设计并实现了一种基于目录可写授权的全异步客户端文件创建框架及其优化技术,测试表明,该技术可将分布式文件系统小文件创建性能提升近500倍。

Massive small files are widely existed in the fields of high -performance computing, corporate office and Internet applications. While mainly stored in distributed file systems, the performance of accessing small files has become a pressing issue in distributed file systems. The key reason for the poor performance is the high proportion of large scale concurrent synchronous access to metadata; too much metadata synchronization can incur lots of network overhead, leading to the delay of accessing small flies, especially in applications which exhibit a high degree of concurrency in creating files in finer granularity. In this paper, based on the pNFS distributed file system, we conduct an in - depth analysis of the key technology of lowering the delay of creating massive small files ; we also design and implement a directory write delegation based, optimized framework with fsully asynchronous creation of files at client side. Test results show that, compared with PNFS, Creation performance may increase nearly 500 times.