期刊文献+

HPMR内存管理模块优化设计

Optimized Design for Memory Management Module in HPMR
下载PDF
导出
摘要 HPMR系统是一个采用MapReduce模型的高性能计算软件支撑平台,它改进了MapReduce模型以适应高性能计算的需求。高效的HPMR系统内存管理模块是保证其效率的重要模块。HPMR系统中有两个角色,Master和Worker。Master负责从输入数据文件中读入数据块并分配给Workers。Worker负责接收master分配的数据块、管理map函数的输入输出模块的输入输出数据、管理reduce函数输入输出数据。目前的内存管理模块存在管理通信冗余、管理低效、数据处理并行不足等缺点。本文根据成熟的内存优化理论,重新设计了HPMR底层的数据管理机制,提出了基于内存池的内存管理。实验表明,新的内存管理模块是保证HPMR系统高效的必要条件。 HPMR is a high performance computing platform based on MapReduce model. It has improved the MapReduce model to meet the need of high performance computing. Efficient memory management module ensures the efficiency of HPMR. There are two roles in HPMR, Master and worker. Master reads data blocks from the input data file and assign them to workers. Worker receives the data blocks from master, manages input and output module of the map and reduce function. The current memory management module in HPMR has some shortcomings: redundancy, inefficiency and lack of parallelism. This paper redesigned the underlying data management mechanism of HPMR based on mature memory optimization theory, proposed new memory management way based on memory pool. Experiments show that the new memory management module is necessary for efficient HPMR system.
出处 《计算机系统应用》 2011年第8期104-109,共6页 Computer Systems & Applications
基金 工信部核高基重大专项(2009X01028-002-003) 安徽省自然科学基金(090412068)
关键词 MAPREDUCE HPMR 内存池 数据管理 MapRoduce HPMR memory pool data management
  • 相关文献

参考文献8

  • 1Dean J, Ghemawat S. MapReduce:simplified data proc. on large clusters. 6th Symposium on Operating Systems Design and Implemention. San Francisco, California, USA, 2004: 1-13.
  • 2Lammel R. Google's MapReduce programming model - revisited. Science of Computer Programming Journal, 2006. 1-42.
  • 3Ranger C, et al. Evaluating MapReduce for Multi-core and Multiproeessor Systems. Proc. of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture, 2007, 13-24.
  • 4郑启龙,王昊,吴晓伟,房明.HPMR:多核集群上的高性能计算支撑平台[J].微电子学与计算机,2008,25(9):21-23. 被引量:11
  • 5Renau J, Strauss K, Ceze L.Thread-Level Speculation on a CMP can be energy efficient. Proc. of the 19th Annual International Conference on Supercomputing, 2005. 219- 228.
  • 6Josuttis NM. The C++ Standard Libaray: A Tutorial and Reference. Addison Wesley Longman, Inc, 1999. 171-190.
  • 7Cormen TH, et al. Introduction to Algorithms. 2nd Ed. The United States: The MIT, 2001.chapter 13.
  • 8Rolando A, Chatvez.C. Natural-convection Heat Transfer in Supercdtical Fluids [MS Thesis]. Mechanical Engineering University of Puerto Rico Mayagtlez Campus, 2004.

二级参考文献2

  • 1Jeffery Dean, Sanjay Ghemawat. MapReduce: simplified data processing on large clusters[ C]//6th Symposium on Operating Systems Design and Implementation. USA, 2004.
  • 2Ralf Laimmel. Google's mapreduce programming model-revisited[ C]//Data Programmability Team Microsoft Corp. USA, Rodmon, 2007.

共引文献10

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部