A Dynamic Memory Allocation Optimization Mechanism Based on Spark

Spark is a distributed data processing framework based on memory.Memory allocation is a focus question of Spark research.A good memory allocation scheme can effectively improve the efficiency of task execution and memory resource utilization of the Spark.Aiming at the memory allocation problem in the Spark2.x version,this paper optimizes the memory allocation strategy by analyzing the Spark memory model,the existing cache replacement algorithms and the memory allocation methods,which is on the basis of minimizing the storage area and allocating the execution area according to the demand.It mainly including two parts:cache replacement optimization and memory allocation optimization.Firstly,in the storage area,the cache replacement algorithm is optimized according to the characteristics of RDD Partition,which is combined with PCA dimension.In this section,the four features of RDD Partition are selected.When the RDD cache is replaced,only two most important features are selected by PCA dimension reduction method each time,thereby ensuring the generalization of the cache replacement strategy.Secondly,the memory allocation strategy of the execution area is optimized according to the memory requirement of Task and the memory space of storage area.In this paper,a series of experiments in Spark on Yarn mode are carried out to verify the effectiveness of the optimization algorithm and improve the cluster performance.

Memory Safety Based on Probabilistic Memory Allocation

Some unsafe languages,like C and C++,let programmers maximize performance but are vulnerable to memory errors which can lead to program crashes and unpredictable behavior.Aiming to solve the problem,traditional memory allocating strategy is improved and a new probabilistic memory allocation technology is presented.By combining random memory allocating algorithm and virtual memory,memory errors are avoided in all probability during software executing.By replacing default memory allocator to manage allocation of heap memory,buffer overflows and dangling pointers are prevented.Experiments show it is better than Diehard of the following aspects:memory errors prevention,performance in memory allocation set and ability of controlling working set.So probabilistic memory allocation is a valid memory errors prevention technology and it can tolerate memory errors and provide probabilistic memory safety effectively.

A research on Android kernel-memory compiling and scheduling

Android, an open source system exploited by Google, has experienced a rapid development in the past a few years in the field of intelligent mobile because of its advantages-open source and excellent function. The number of professionals and enthusiasts who research on Android is growing rapidly in the same time. Android, as an abstraction between software layer and hardware layer based on Linux kernel, can complete the optimization of system by modifying the kernel part. The purpose of this design is to master the processes of kernel-compiling and transplanting, and to learn the methods of memory scheduling algorithm and kernel menaory test. First of all, this thesis introduces the installation of Linux system, and then, it presents the method to build the environment for Android kernel compiling and the process of compiling. The key point of the design is to introduce the SLAB, SLOB, SLUB, SLQB allocators in memory scheduling, and carry on a research on optimization with these memory allocators. HTC Incredible S, as an experimental mobile phone whose Android kernel version is 2.6.35, is employed to deal with all these tests. A comparison of kernel codes before and after optimization has been made. The two kernel codes have been transplanted into the terminal of the experimental mobile phone, which will be respectively tested with its stability, memory performance and overall performance. Finally, it concludes that result of being transplanted the SLQB memory allocator is the optimal one of all.

Exploitation of symmetry in periodic Self-Consistent-Field ab initio calculations: application to large three-dimensional compounds

Symmetry can dramatically reduce the computational cost （running time and memory allocation） of Self-Consistent-Field ab initio calculations for crystalline systems. Crucial for running time is use of symmetry in the evaluation of one- and two-electron integrals, diagonalization of the Fock matrix at selected points in reciprocal space, reconstruction of the density matrix. As regards memory allocation, full square matrices （overlap, Fock and density） in the Atomic Orbital （AO） basis are avoided and a direct transformation from the packed AO to the SACO （Symmetry Adapted Crystalline Orbital） basis is per- formed, so that the largest matrix to be handled has the size of the largest sub-block in the latter basis. We here illustrate the effectiveness of this scheme, following recent advancements in the CRYSTAL code, concerning memory allocation and direct basis set transformation. Quantitative examples are given for large unit cell systems, such as zeolites （all-silica faujasite and silicalite MF1） and garnets （pyrope）. It is shown that the full SCF of 3D systems containing up to 576 atoms and 11136 Atomic Orbitals in the cell can be run with a hybrid functional on a single core PC with 500 MB RAM in about 8 h.