面向I/O密集型应用的分离执行模型的实现与优化

Implement and Optimization of the Decoupled Execution for I/O Intensive Applications

在众多科学大数据计算应用中,I/O性能已逐渐成为制约应用程序性能的主要瓶颈.然而随着硬件设备的更新与发展,现有高性能计算系统的计算性能与I/O性能差距逐渐增大.传统的以计算为中心的执行模型旨在利用内存和CPU的性能来解决I/O瓶颈问题.本文面向I/O密集型的科学大数据应用,实现并优化了新的执行模型--分离执行模型.该模型通过分离应用的计算操作与I/O操作,对I/O操作予以统一的管理和调度,以此来解决当前HPC系统上具有挑战性的应用I/O瓶颈问题.分离执行模型在架构上增加专门负责I/O操作的中间节点;在实现上利用消息传递模型(MPI)的I/O操作接口,重新定义并划分I/O操作流程.实验表明,该模型有效减少数据在网络中的传输,加快计算过程对数据的访问,从而提升10%至20%的I/O性能.其以数据为中心的架构模型思想对我国下一代高性能超级计算机系统结构研发设计具有参考意义.

In many scientific big-data computing applications,I/O performance has become a vital bottleneck restricting the performance of the applications.However,with the update and development of the hardware equipment,the gap between the computing performance and I/O performance increases gradually in the existing high-performance computing system.The conventional execution models are computing-centric for computation.They are designed to solve the I/O bottleneck issues by utilizing memory and CPU performance.In this paper,wo implement and optimize the decoupled execution model for these big-data computing applications.This model solves the challenging I/O bottleneck issues in the HPC systems by separating the computing operations and I/O operations of the application and having unified management and scheduling of the I/O operations.The decoupled execution model adds data nodes to conduct the I/O operations on the architecture.And this model redistributes the procedure of I/O operations by imitating the interface of the message passing interface(MPI).The experiment shows that this model can significantly accelerate the computing process to access the data by reducing costly data movement on the interconnect,and improves the I/O performance by 10%-20%.This datacentric architecture model has reference significance for the design of the next generations high-performance supercomputer architecture in our country.