面向分层混合存储架构的协同式突发缓冲技术

Research on optimization towards hybrid and hierarchy storage architecture

科学计算产生和分析的数据规模日益增长,高性能计算机的存储系统在体系架构和软件管理方法上面临重大挑战。针对天河-2系统的新型分层混合存储架构,提出一种由应用程序耦合的协同式突发缓冲技术来有效利用其存储资源优势。该方法采用运行时动态耦合的方法,将临近计算任务的分布式高速存储资源聚合成为一个的巨大的协同式突发缓冲区,通过采用文件命名空间投影的映射方法组织全局数据视图,利用位置亲和和感知数据意图的方法来挖掘空间局部性和时间局部性,并利用应用并发度感知的策略优化数据移动效率。天河-2系统的测试结果表明,该方法能够有效优化多种典型应用场景,可获得高可扩展的突发并行输出带宽和稳定的持续并行输出带宽,可显著提升数据分析场景的输入/输出性能,适合应用于大规模超级计算机的存储系统。

Today＇s advancing modern science is generating and analyzing increasing scale of datasets and makes HPC storage system facing new challenges both on architecture and software approach. In order to exploit potential benefits of emerging hybrid and hierarchy storage architecture on Milky- 2 system,a I / O middleware approach named application coupled burst buffer,is introduced to make full use of the solid state disk based in-system storage resources. Application coupled burst buffer aggregates distributed in-system storage close to running tasks into single namespace during application runtime and manages it as cooperative persistent burst buffer tightly coupled with its host application. To take full advantage of cooperative burst buffer,application coupled burst buffer uses an unified shadow namespace to map application data into physical in-system storage based on its real namespace of the host application. Besides that,application coupled burst buffer organizes data with locality aware layout and leverages application intent based replacement policy to fully exploit spatial and temporal locality. Furthermore,application coupled burst buffer employs concurrency aware policies to optimize data movement between different storage tiers. Evaluations on Milky- 2 system show that application coupled burst buffer can improve the performance of typical data-intensive applications dramatically. It can achieve scalable burst I / O bandwidth and smooth sustained I / O bandwidth with high throughput solid state disk deployed and can be taken as an appropriate candidate for storage solution on emerging leadership supercomputer systems.