How Big Data and High-performance Computing Drive Brain Science

Brain science accelerates the study of intelligence and behavior,contributes fundamental insights into human cognition,and offers prospective treatments for brain disease.Faced with the challenges posed by imaging technologies and deep learning computational models,big data and high-performance computing(HPC)play essential roles in studying brain function,brain diseases,and large-scale brain models or connectomes.We review the driving forces behind big data and HPC methods applied to brain science,including deep learning,powerful data analysis capabilities,and computational performance solutions,each of which can be used to improve diagnostic accuracy and research output.This work reinforces predictions that big data and HPC will continue to improve brain science by making ultrahigh-performance analysis possible,by improving data standardization and sharing,and by providing new neuromorphic insights.

面向高性能应用的MPI大数据处理

针对消息传递接口(MPI)在高性能计算领域的应用场景,为了优化MPI现有数据集中管理模式,增强其对大数据的处理能力,借鉴并行与分布式系统思想,开发设计一套适用于大数据处理的基于MPI的数据存储组件(MPI-DSP)。首先,创建接口函数,以对MPI系统影响最小的方式实现"计算向存储迁移"的设计目标,将文件分配与计算进行分离,使MPI突破大数据文件读取时的网络传输瓶颈。然后,分析阐述设计目标、运行机制、实现策略,通过描述接口函数MPI_Open在MPI环境下的应用,验证设计理念。通过Wordcount实验对比使用MPI-DSP组件与原MPI在数据文件处理方面的时间性能,初步验证了MPI"计算向存储迁移"模式的可行性,使其具备在高性能应用场景下的大数据处理能力。同时分析了MPI-DSP的适用环境和局限性,界定了其应用范围。

A geospatial hybrid cloud platform based on multi-sourced computing and model resources for geosciences

Cloud computing has been considered as the next-generation computing platform with the potential to address the data and computing challenges in geosciences.However,only a limited number of geoscientists have been adapting this platform for their scientific research mainly due to two barriers:1)selecting an appropriate cloud platform for a specific application could be challenging,as various cloud services are available and 2)existing general cloud platforms are not designed to support geoscience applications,algorithms and models.To tackle such barriers,this research aims to design a hybrid cloud computing(HCC)platform that can utilize and integrate the computing resources across different organizations to build a unified geospatial cloud computing platform.This platform can manage different types of underlying cloud infrastructure(e.g.,private or public clouds),and enables geoscientists to test and leverage the cloud capabilities through a web interface.Additionally,the platform also provides different geospatial cloud services,such as workflow as a service,on the top of common cloud services(e.g.,infrastructure as a service)provided by general cloud platforms.Therefore,geoscientists can easily create a model workflow by recruiting the needed models for a geospatial application or task on the fly.A HCC prototype is developed and dust storm simulation is used to demonstrate the capability and feasibility of such platform in facilitating geosciences by leveraging across-organization computing and model resources.

面向E级计算超融合软件框架的设计与实现

从高性能计算机体系结构上看,其正朝着超融合、多态复合、自适应方向发展,同时节点异构仍将是未来顶级高性能计算机的主流,而计算体系结构与应用适配的问题日渐突出,集群资源利用率低成为一个亟待解决的问题。介绍了一种面向E级计算超融合软件框架的设计与实现,系统融合高性能计算、深度学习、大数据以及云计算等应用处理技术,构建两级资源调度机制,解耦资源调度和资源分配,柔性扩展异构应用;并通过轻量级容器技术封装应用和隔离运行环境,实现应用动态扩展与部署管理。基于开放性协议实现整个集群资源的数据采集和分析处理,对集群提供智能数据分析依据,提高系统资源利用率,促进应用高效运行。

Design and Implementation of the Tianhe-2 Data Storage and Management System

With the convergence of high-performance computing(HPC),big data and artificial intelligence(AI),the HPC community is pushing for"triple use"systems to expedite scientific discoveries.However,supporting these converged applications on HPC systems presents formidable challenges in terms of storage and data management due to the explosive growth of scientific data and the fundamental differences in I/O characteristics among HPC,big data and AI workloads.In this paper,we discuss the driving force behind the converging trend,highlight three data management challenges,and summarize our efforts in addressing these data management challenges on a typical HPC system at the parallel file system,data management middleware,and user application levels.As HPC systems are approaching the border of exascale computing,this paper sheds light on how to enable application-driven data management as a preliminary step toward the deep convergence of exascale computing ecosystems,big data,and AI.

Networking and communication challenges for post-exascale systems

With the significant advancement in emerging processor, memory, and networking technologies, exascale systems will become available in the next few years （2020 2022）. As the exascale systems begin to be deployed and used, there will be a continuous demand to run next-generation applications with finer granularity, finer time-steps, and increased data sizes. Based on historical trends, next-generation applications will require post-exascale systems during 2025-2035. In this study, we focus on the networking and communication challenges for post-exascale systems. Firstly, we present an envisioned architecture for post-exascale systems. Secondly, the challenges are summarized from different perspectives： heterogeneous networking technologies, high-performance eonmmnication and synchronization protocols, integrated support with accelerators and field-programmable gate arrays, fault-tolerance and quality-of-service support, energy-aware communication schemes and protocols, software- defined networking, and scalable communication protocols with heterogeneous memory and storage. Thirdly, we present the challenges in designing efficient programming model support for high-performance computing, big data, and deep learning on these systems. Finally, we emphasize the critical need for co-designing runtime with upper layers on these systems to achieve the maximum performance and scalability.