梯度法简述

A brief review on gradient method

梯度法是一类求解优化问题的一阶方法。梯度法形式简单、计算开销小,在大规模问题的求解中得到了广泛应用。系统地介绍了光滑无约束问题梯度法的迭代格式、理论框架。梯度法中最重要的参数是步长,步长的选取直接决定了梯度法的收敛性质与收敛速度。从线搜索框架、近似技巧、随机技巧和交替重复步长四方面介绍了梯度步长的构造思想及相应梯度法的收敛性结果,还对非光滑及约束问题的梯度法、梯度法加速技巧和随机梯度法等扩展方向做了简要介绍。

Gradient method is a kind of first order optimization method.It is widely used for large scale problems,due to its simplicity and low complexity.This paper is a review on gradient method.Gradient methods for smooth unconstrained problems are introduced,with details of algorithm framework and theories.The crucial factor in gradient method is the stepsize,which determines the convergence property of the method.This paper reviews the stepsize update strategies and the corresponding convergence results from four aspects:line search,approximation technique,stochastic technique,alternating and constant stepsizes.Other related topics like gradient method for nonsmooth and constrained optimization problems,acceleration technique and stochastic gradient method are also mentioned.