一种赋有BB类步长的新随机方差缩减梯度算法

A New Stochastic Variance Reduced Gradient Algorithm with BB-Like Step Size

为随机方差缩减梯度(stochastic variance reduced gradient,SVRG)算法引入自适应步长,并在此基础上进一步提高算法数值性能。首先利用具有二维二次终止性的BB类步长自适应计算SVRG算法的步长。然后在SVRG算法的内循环中引入停止准则和负动量来加速算法的收敛速度。利用Matlab对提出的新算法进行数值实验,观察算法的数值性能。通过分析算法的数值实验结果,得出算法性能与在最佳步长调整下的SVRG算法相当,此外新算法对于初始步长的选取不敏感,且具有自动生成最优步长的能力。

Adaptive step size is introduced into the Stochastic Variance Reduction Gradient(SVRG)algorithm,and the numerical performance of the algorithm is further improved on this basis.Firstly,the step size of the SVRG algorithm is calculated by BB step size with two-dimensional quadratic termination.Then the stopping criterion and negative momentum are introduced into the inner loop of the SVRG algorithm to accelerate the convergence rate.The proposed algorithm's numerical experiment was carried out using Matlab,and the numerical performance of the algorithm was observed.By analyzing the numerical experimental results of the algorithm,it is concluded that the algorithm's performance is comparable to that of the SVRG method with the optimal step size adjustment.In addition,the new algorithm is insensitive to the selection of the initial step size and can automatically generate the optimal step size.