摘要
随着大数据应用和传统高性能计算应用的融合以及异构计算的引入,传统面向高性能计算的并行存储系统面临着异构计算I/O支持差、性能干扰和效率低等问题。通过在系统架构引入多层次存储架构、设计缓存映射机制来减轻I/O负载。在转发服务层,调整I/O转发策略,均衡I/O负载。在后端存储层,对系统高可用功能进行调整,解决大数据I/O访问模式与原有高可用措施的冲突。经过优化设计和完善后的并行存储系统更好地适应了异构众核架构,使得某些应用获得了10倍以上的I/O性能提升。
With the integration of big data applications and traditional high-performance computing applications and the introduction of heterogeneous computing,the traditional parallel storage system for high-performance computing faces the problems of poor I/O support,performance interference,and low efficiency.By introducing multi-level storage architecture into the system architecture,the cache mapping mechanism was designed to reduce the I/O load.The I/O forwarding strategy was adjusted in the forwarding service layer to balance the I/O load.In the back-end storage layer,the high availability function of the system was adjusted to solve the conflict between the big data I/O access mode and the original high availability functions.After optimized design and improvement,the parallel storage system can better adapt to the heterogeneous multi-core architecture,making some applications get more than 10 times of I/O performance improvement.
作者
何晓斌
蒋金虎
HE Xiaobin;JIANG Jinhu(National Parallel Computing Engineering Technology Research Center,Beijing 100080,China;School of Computer Science and Technology,Fudan University,Shanghai 200433,China)
出处
《大数据》
2020年第4期30-39,共10页
Big Data Research
关键词
大数据
高性能计算
神威·太湖之光
异构
并行存储
big data
high performance computing
Sunway TaihuLight
heterogeneous
parallel storage
