并行计算框架的内存优化算法研究

Distributed independent memory allocation algorithms of parallel computing framework

传统的大数据内存分配算法存在运行速度慢、分配不均匀的问题,为此,提出一种新的并行计算框架的内存优化算法。通过利用无监督贪婪模式逐层训练方法进行训练学习并建立并行框架,采用分布式存储的方法对数据进行承载,最大程度提升运算性能。同时,对较小内存Task做分化处理,保证算法的执行效率,并且会避免不必要的溢出操作。实验结果证明,相比于传统算法,所提算法的内存分配情况更合理、运行速度快,内存分配效果更好。

The traditional big data memory allocation algorithm has the problems of slow running speed and uneven allocation.Therefore,a new memory optimization algorithm of parallel computing framework is proposed.Through useing unsupervised greedy model layer by layer training method for training and learning,establishing a parallel framework,the method of distributed storage can carry out data to maximize the performance of the operation.At the same time,the small memory tasks are differentiated to ensure the efficiency of the algorithm and avoid unnecessary overflow operations.The experimental results show that compared with the traditional algorithm,the proposed algorithm has more reasonable memory allocation,faster running speed and better memory allocation effect.