面向国家高性能计算环境的虚拟数据空间系统

Virtual data space system for national highperformance computing environment

高性能计算环境是支撑国家科技创新、经济发展、国防建设的核心信息基础设施,世界高性能计算强国纷纷建设基于多超算中心资源的广域高性能计算环境。然而,高性能计算环境中资源种类繁多且地域分布广,无法有效发挥资源的聚合效应,难以满足大型应用对广域分布数据的统一管理和高效访问需求。为此,提出了一套可用于构建广域全局虚拟数据空间的完整技术体系,包括虚拟数据空间模型、跨域虚拟数据空间构建、广域环境中数据高效迁移、广域环境中存算协同调度、跨域高并发数据聚合处理等技术,并研发了一个可运行于国家高性能计算环境的虚拟数据空间系统,可有效支撑广域分散异构存储资源的统一高效访问,实现广域环境中分布数据的跨域共享和协同处理。目前,该软件系统已在国家高性能计算环境实验性部署,并验证了分子对接、全基因组关联分析、天气预报模式3类典型大型应用。验证结果表明,所研虚拟数据空间构建方法和系统可有效聚合广域分散的存储资源,满足大型应用的数据空间需求。

High-performance computing(HPC)environment is the core information infrastructure supporting national scientific and technological innovation,economic development and national defense construction.High-performance computing powers around the world have been building wide-area HPC environments based on multi-supercomputing center resources.However,in the high-performance computing environment,there are many kinds of resources and wide geographical distribution,which cannot effectively exert the aggregation effect of resources,and it is difficult to meet the requirements of large-scale applications for unified management and efficient access to wide-area distributed data.To this end,a complete set of technologies were proposed,which could be used to build wide-area global virtual data space,including virtual data space model,cross-domain virtual data space constructing,efficiently migrating data in a wide-area environment,co-scheduling of storage resources and computing job and cross-domain high concurrency data aggregation processing,etc.Based on the above,a virtual data space system has been developed for the national high-performance computing environment(NHPCE),which can effectively support the unified and efficient access to the wide area distributed heterogeneous storage resources,and the distributed data in the wide-area environment can be shared and cooperative processed in a cross-domain manner.At present,the system was experimental deployed in NHPCE and three typical large-scale applications,such as molecular docking,genome-wide association study and weather forecasting model,have been verified.The verification results show that the developed technology and software system can effectively aggregate the wide area distributed storage resources and meet the data space requirements of large-scale applications.