Barzilai-Borwein型算法的探讨

Discussion on Barzilai-Borwein Algorithm

本文介绍了八种负梯度算法,根据特点对其进行了比较,并对不同维数的严格凸二次函数进行了计算,绘制图表观察数据,发现BB型算法更具优势,维数越大优势越大。选取不同的初始步长可以改变算法的效果,特别是当矩阵条件数越大时,初始步长的选取越关键,本文考虑了四种选择初始步长的方法,分别将其放入算法中进行数值实验,结果表明,选取Hessian矩阵最小特征值的倒数效果最好。最后,介绍了负梯度算法在深度学习中的应用。

This paper introduces eight kinds of negative gradient algorithms, compares them according to their characteristics, calculates strictly convex quadratic functions of different dimensions, draws graphs and observes data, and finds that BB algorithm is more advantageous, the larger the dimension is, the greater the advantage is. Selecting different initial step size can change the effect of the algorithm, especially when the number of matrix conditions is larger, the selection of the initial step size is more critical. This paper considers four methods of selecting the initial step size, and puts them into the algorithm for numerical experiments. The results show that the reciprocal of the minimum eigenvalue of the Hessian matrix is the best. Finally, the application of negative gradient algorithm in deep learning is introduced.