结合算子选择的卷积神经网络显存优化算法

Memory Optimization Algorithm for Convolutional Neural Networks with Operator Selection

针对卷积神经网络训练中自动算子选择算法在较大的显存压力下性能下降的问题,将卸载、重计算与卷积算子选择统一建模,提出一种智能算子选择算法。该算法权衡卸载和重计算引入的时间开销与更快的卷积算子节省的时间,寻找卸载、重计算和卷积算子选择的调度,解决了自动算子选择算法性能下降的问题.实验结果表明,该智能算子选择算法比重计算-自动算子选择算法缩短了13.53%训练时间,比已有的卸载/重计算-自动算子选择算法缩短了4.36%的训练时间.

Aiming at the problem of the performance degradation of the automatic operator selection algorithm in convolutional neural network training under high memory pressure,we modelled offloading,recomputing and convolutional operator selecting in a unified manner and proposed an intelligent operator selection algorithm.The algorithm weighed the time overhead introduced by offloading and recomputing against the time saved by faster convolutional operators,found the[JP+1]scheduling of offloading,recomputing and convolutional operator selecting,and solved the performance[JP]degradation problem of the automatic operator selection algorithm.The experimental results show that the intelligent operator selection algorithm reduces training time by 13.53%over the recomputing-automatic operator selection algorithm and by 4.36%over the existing offloading/recomputing-automatic operator selection algorithm.