面向大规模集群的并行I/O用户层配置优化策略

User-level parallel I/O configuration optimize strategy toward large-scale cluster

影响应用I/O性能的关键因素主要有三个层次:包括应用的I/O接口实现、体系结构和文件系统组件的性能以及应用的I/O参数配置。从应用I/O配置优化的视角,分析了大规模集群并行I/O的配置调优空间,在此基础上,给出了一套大规模集群并行I/O性能特征测试分析方法。基于该方法,在某国产超级计算集群上开展了一系列I/O测试分析来刻画系统的I/O性能特征,进而指导并行应用程序的I/O配置优化。基于优化后的配置参数,在两类典型的并行I/O场景中,针对某类生产应用程序,8192进程下的重启动数据写操作时间下降了15%,4096核的程序作业加载时间从10 min缩短到了5 s。

Three key factors exert big influence upon the application′s I/O performance,including the I/O programming interface,the performance characteristic of I/O sub-system(both architecture and system software),and the I/O configuration parameters at user-level.From the user′s perspective,this paper discussed the user-level parallel I/O configuration optimize space toward large scale cluster.Besides,we proposed a method of testing and analyzing the I/O characteristic of large scale cluster.Based on this method,the I/O performance portrait of a domestic super computer was built up and several user-level parallel I/O optimize suggestions were put forward.With these carefully selected I/O configuration parameters,the time of restart data write operation was cut down by 15 percent under 8192 processes in a real application environment,while the program′s initial time is shortened from 10 minutes to 5 seconds at the scale of 4096 processes.