并行计算框架Spark的自动检查点策略

Automatic checkpoint strategy for parallel computing frame with Spark

针对现有的Spark检查点机制需要编程人员根据经验选择检查点,具有一定的风险和随机性,可能导致恢复开销较大的问题,通过对RDD属性的分析,提出了自动检查点策略,包括权重生成(WG)算法和检查点自动选择(CAS)算法.首先,WG算法分析作业的DAG结构,获取RDD的血统长度和操作复杂度等属性,计算RDD权重;然后,CAS算法选择权重大的RDD作为检查点进行异步备份,来实现数据的快速恢复.结果表明:在使用CAS算法时,不同数据集执行时间和检查点容量大小都有所增加,其中Wiki-Talk由于其计算量较大,增幅明显;使用CAS算法设置检查点后,在单点失效恢复的情况下,数据集的恢复时间较短.因此,自动检查点策略在略微增加执行时间开销的基础上,能够有效地降低作业的恢复开销.

The existing Spark checkpoint mechanism required the programmer to choose the checkpoint according to the experience,thus it had a certain risk and randomness,resulting in large recovery overhead. To address this problem,the resilient distribution datasets（ RDD） characteristics were analyzed,and the weight generated（ WG） algorithm and checkpoint automatic selection（ CAS） algorithm were put forward. First,in the WG algorithm,the directed acyclic graph（ DAG） of the job was analyzed,and the lineage length and the operation complexity of RDD were obtained to compute the RDD weight. Secondly,in the CAS algorithm,the RDD with the maximum weight was selected for setting checkpoints asynchronously to fast recovery. The experimental results showthat comparing with the original Spark,the execution time and the checkpoint size of different datasets are increased by the CAS algorithm,while the increasing extent of Wiki-Talk is more obvious. For the single node failure recovery,the datasets have smaller recovery overhead after setting checkpoint by using the CAS algorithm. Therefore,the strategy can efficiently decrease the recovery overhead of jobs with sacrificing the slight extra overhead.