无约束优化之线性搜索技术研究

Research on the Linear Search Algorithm with Unconstrained Optimization

梯度下降法及其变体是最常用的数值优化算法之一,也是迄今为止优化神经网络最常用的方法.在每一个最新的深度学习库中几乎都包含了各种优化的梯度下降法的实现.作为其关键子问题的线性搜索技术解决了梯度下降法带来的收敛速度慢、易陷入局部极值等缺点,实现了非线性函数求全局极值的快速收敛.本文对线性搜索技术及其收敛性进行深入研究,实现了基于Armijo条件的回溯算法,并对其性能进行了分析.

Gradient descent method and its variants are one of the most commonly used numerical optimization algorithms,and the most commonly used methods to optimize neural networks so far.In each of the latest depth learning database,almost all of the optimized gradient descent method are implemented.The convergence rate of gradient descent method is slow and easy to fall into the local extremum,however,as its sub-problem,linear search algorithm realizes the fast convergence of the nonlinear function for global extremum.This paper aims at further discussing the linear search technology and its convergence to help readers understand and use such algorithms correctly and implemehts a backtracking algorithm based on Armijo conditions and analyzes its performace.