无约束优化一阶算法的几类加速技巧

SEVERAL KINDS OF ACCELERATION TECHNIQUES FOR UNCONSTRAINED OPTIMIZATION FIRST-ORDER ALGORITHMS

求解无约束优化问题的一阶算法具有迭代简单、存储量小的优点,在求解大规模问题时具有一定的优势.为提升其收敛速度,近些年发展出了多种加速技巧.本文以最一般的求解无约束优化的梯度法为切入点,介绍常见的加速梯度法的技巧与策略,并进一步介绍这些加速技巧在邻近点算法、复合优化问题和随机优化问题中的表现形式.另外,本文还总结了一些其它仅用一阶信息就取得加速效果的策略和特殊问题中出现的加速方法.

The first-order algorithms provide several advantages in tackling large-scale problems,as well as the benefits of simple iteration and little storage.To speed up the convergence,numerous acceleration strategies have been created recently.The gradient method of unconstrained optimization is used as the starting point for this paper,and the common techniques and strategies of the accelerated gradient method are also introduced.These acceleration techniques are further explained in terms of the expressions in the proximal point algorithm,composite optimization problem and stochastic optimization problem.Moreover,this paper provides a summary of additional ways for acceleration strategies using only first-order information and acceleration techniques that are utilized in specific problems.