基于Alluxio的异步存储优化

Asynchronous storage optimization based on Alluxio

为了减缓与底层存储直接传输数据的网络压力,保证异步传输速度与数据可靠性,综合分析了生成文件的操作是否可重算和重算时间等条件,采用传数据和传操作相结合的方式进行数据持久化.相对于传数据,传操作利用底层存储的计算资源,以较小数据传输代价完成部分数据的持久化.对于无法重计算恢复的文件,采用同步与异步相结合的策略保证文件的可靠性.实验结果表明,结合文件操作的异步存储策略Async-Store的运行时间比单纯的同步策略缩短41%,并在一定程度上保证了数据的可靠性.相比异步策略,同步与异步相结合的策略Async&Sync通过牺牲少量性能完全保证数据的可靠性,其运行时间较同步策略缩短26%.

In order to reduce the work load of directly transmitting data to the underlying storages as well as guarantee the speed of asynchronous data transmitting and the reliability of data,the constraints of data transmission and sending operation such as the operation of generating a file is recalculated or not and the recomputation time of a file are comprehensively analyzed,and the method by combining sending data with sending operation is proposed for data persistence.Compared with the transmitting data,sending operation can reduce the work load of network transmission by using the computing resources of underlying storages.To the files which cannot be recovered by recomputation,a strategy by combining asynchronization with synchronization(Async&Sync)is adopted to ensure the reliability of the file.The experimental results show that compared with the synchronization strategy,the running time of the asynchronous strategy combined with file operation can be reduced by 41%and the reliability of the data in some degree is guaranteed.Compared with the asynchronous strategy,the Async&Sync strategy can completely guarantee the reliability of the data with lower performance costs.And the running time is reduced by 26%compared with the synchronization strategy.