一种修正学习率的梯度下降算法

A Gradient Descent Algorithm with Modified Learning Rate

近年来,随着神经网络的广泛应用,梯度下降算法成为神经网络调整参数的核心优化算法。随着SGDM、AdaGrad、RMPprop、Adam等算法被提出,神经网络的性能有了极大的提升。为改善Adam算法由极端学习率引起的泛化能力较差问题,利用梯度的指数加权平均对Adam算法中学习率进行修正,提出了MonAdam算法。通过在线学习框架,说明MonAdam算法具有O(√T)的遗憾界。经过大量实验在多种非凸函数和深度神经网络模型中将MonAdam算法与其他算法进行对比,结果表明该算法收敛性较好。

In recent years,with the wide application of neural networks,the gradient descent algorithm has become the core optimization algorithm for neural network adjustment parameters.With the introduction of SGDM,AdaGrad,RMPprop,Adam,etc.,the performance of neural networks has been greatly improved.In order to improve the poor generalization ability of Adam algorithm caused by extreme learning rate,the exponential weighted average of gradient is used to correct the learning rate in Adam,and MonAdam is proposed.Through the online learning framework,it shows that MonAdam has the regret bound of O(√T).After a large number of experiments,MonAdam is compared with other algorithms in a variety of non-convex functions and deep neural network models.The results show that the algorithm has better convergence.