深度学习中的预条件动量梯度算法

Preconditioned momentum gradient algorithm in deep learning

结合“比例积分微分”优化控制器(Proportional-Integral-Derivative Controller,PID)与拟牛顿方法,提出一种加速神经网络训练的一类预条件动量梯度算法。首先,采用拟牛顿条件产生预条件因子,然后,在迭代过程中与动量方法相结合,提高算法效率的同时,克服原始动量梯度算法的超调问题,并证明了算法在目标函数强凸时的全局收敛性。最后,通过数值实验验证了算法的有效性。

Deep learning usually requires solving large-scale optimization problem.Based on the traditional Proportional-Integral-Derivative Controller(PID),combined with the Quasi-Newton Method,this paper proposes a kind of preconditioned momentum algorithm to accelerate neural network training.It can effectively overcome the overshoot phenomenon of the original momentum method.Moreover,it can also improve greatly the efficiency of the algorithm due to the usage of the preconditioning factors.When the objective function is strongly convex,the global convergence of the algorithm can be guaranteed.Finally,numerical experiments verify the effectiveness of the proposed algorithm.