A Dynamic Memory Allocation Optimization Mechanism Based on Spark

Spark is a distributed data processing framework based on memory.Memory allocation is a focus question of Spark research.A good memory allocation scheme can effectively improve the efficiency of task execution and memory resource utilization of the Spark.Aiming at the memory allocation problem in the Spark2.x version,this paper optimizes the memory allocation strategy by analyzing the Spark memory model,the existing cache replacement algorithms and the memory allocation methods,which is on the basis of minimizing the storage area and allocating the execution area according to the demand.It mainly including two parts:cache replacement optimization and memory allocation optimization.Firstly,in the storage area,the cache replacement algorithm is optimized according to the characteristics of RDD Partition,which is combined with PCA dimension.In this section,the four features of RDD Partition are selected.When the RDD cache is replaced,only two most important features are selected by PCA dimension reduction method each time,thereby ensuring the generalization of the cache replacement strategy.Secondly,the memory allocation strategy of the execution area is optimized according to the memory requirement of Task and the memory space of storage area.In this paper,a series of experiments in Spark on Yarn mode are carried out to verify the effectiveness of the optimization algorithm and improve the cluster performance.

KDS-CM:A Cache Mechanism Based on Top-K Data Source for Deep Web Query

Caching is an important technique to enhance the efficiency of query processing. Unfortunately, traditional caching mechanisms are not efficient for deep Web because of storage space and dynamic maintenance limitations. In this paper, we present on providing a cache mechanism based on Top-K data source （KDS-CM） instead of result records for deep Web query. By integrating techniques from IR and Top-K, a data reorganization strategy is presented to model KDS-CM. Also some measures about cache management and optimization are proposed to improve the performances of cache effectively. Experimental results show the benefits of KDS-CM in execution cost and dynamic maintenance when compared with various alternate strategies.

Enabling Efficient Caching in High Mobility UAV Communications Network under Limited Backhaul

Due to flexible deployment,unmanned aerial vehicle(UAV)mounted aerial access points are capable of expanding the coverage capabilities of existing terrestrial base stations(TBSs).Different from TBSs,however,UAV access points(UAPs)are of high mobility in horizontal and vertical dimensions,which may deteriorate the coverage performance.Worsestill,the mobility of UAPs would as well increase the pressure of wireless backhaul.In this light,we investigate the performance of the cache-enabled UAV communications network(CUCN)in terms of network spatial throughput(ST)by analyzing the line of sight(LoS)connections and non-line of sight(NLoS)connections.It is found that the network ST is exponentially decreased with the square of UAP altitude.Furthermore,contrary to intuition,a large cache size may deteriorate the network ST when UAPs are over-deployed.The reason is that a large cache size increases the hit probability,which may increase the activation of UAPs and consequently result in complicated interference.Aiming to maximize the network ST,we optimize the cache strategy under limited backhaul.Remarkably,the results show that network ST could be substantially improved by the optimized cache strategy and the performance degeneration brought by UAP high mobility could be even eliminated especially when the UAP altitude is high.