摘要
近年来,随着神经网络的广泛应用,梯度下降算法成为神经网络调整参数的核心优化算法。随着SGDM、AdaGrad、RMPprop、Adam等算法被提出,神经网络的性能有了极大的提升。为改善Adam算法由极端学习率引起的泛化能力较差问题,利用梯度的指数加权平均对Adam算法中学习率进行修正,提出了MonAdam算法。通过在线学习框架,说明MonAdam算法具有O(√T)的遗憾界。经过大量实验在多种非凸函数和深度神经网络模型中将MonAdam算法与其他算法进行对比,结果表明该算法收敛性较好。
In recent years,with the wide application of neural networks,the gradient descent algorithm has become the core optimization algorithm for neural network adjustment parameters.With the introduction of SGDM,AdaGrad,RMPprop,Adam,etc.,the performance of neural networks has been greatly improved.In order to improve the poor generalization ability of Adam algorithm caused by extreme learning rate,the exponential weighted average of gradient is used to correct the learning rate in Adam,and MonAdam is proposed.Through the online learning framework,it shows that MonAdam has the regret bound of O(√T).After a large number of experiments,MonAdam is compared with other algorithms in a variety of non-convex functions and deep neural network models.The results show that the algorithm has better convergence.
作者
姜文翰
姜志侠
孙雪莲
JIANG Wenhan;JIANG Zhixia;SUN Xueian(School of Mathematics and Statistics,Changchun University of Science and Technology,Changchun 130022)
出处
《长春理工大学学报(自然科学版)》
2023年第6期112-120,共9页
Journal of Changchun University of Science and Technology(Natural Science Edition)
基金
国家自然科学基金(11426045)
吉林省自然科学基金(YDZJ202201ZYTS519,YDZJ202201ZYTS585)。
关键词
梯度下降算法
Adam算法
收敛性
遗憾界
学习率
gradient descent algorithm
Adam algorithm
convergence
regret bound
learning rate