用于训练神经网络的自适应梯度下降优化算法

Adaptive gradient descent optimization algorithm for training neural networks

由于神经网络规模的扩大,模型训练变得越来越困难.为应对这一问题,提出了一种新的自适应优化算法——Adaboundinject.选取Adam的改进算法Adabound算法,引入动态学习率边界,实现了自适应算法向随机梯度下降(SGD)的平稳过渡.为了避免最小值的超调,减少在最小值附近的振荡,在Adabound的二阶矩中加入一阶矩,利用短期参数更新作为权重,以控制参数更新.为了验证算法性能,在凸环境下,通过理论证明了Adaboundinject具有收敛性.在非凸环境下,进行了多组实验,采用了不同的神经网络模型,通过与其他自适应算法对比,验证了该算法相比其他优化算法具有更好的性能.实验结果表明,Adaboundinject算法在深度学习优化领域具有重要的应用价值,能够有效提高模型训练的效率和精度.

Due to the increasing scale of neural networks,model training had become increasingly challenging.In response to this issue,a new adaptive optimization algorithm called Adaboundinject was proposed.Building upon the improved Adam algorithm,the Adabound algorithm introduced dynamic learning rate boundaries to facilitate a smooth transition from adaptive optimization to stochastic gradient descent(SGD).To avoid overshooting the minimum value and reduce oscillations near the minimum,a first moment was introduced into the second moment of Adabound,utilizing short-term parameter updates as weights to control parameter updates.To validate the algorithm s performance,its convergence properties were theoretically proved in a convex environment.In a non-convex environment,multiple experiments were conducted using different neural network models,comparing the algorithm with other adaptive algorithms,and demonstrating its superior performance.The experimental results indicated that the Adaboundinject algorithm held significant value in the field of deep learning optimization,effectively enhancing both the efficiency and accuracy of model training.