基于随机L-BFGS的二阶非凸稀疏优化算法

Second-Order Nonconvex Sparse Optimization Method Based on Stochastic L-BFGS

在稀疏模型中普遍采用一阶优化算法进行学习,这些算法的普遍思路都是将迭代硬阈值算法与优化算法进行结合。相较于一阶优化算法,二阶优化算法很少被应用到稀疏优化问题中,因为Hessian矩阵及其逆矩阵的计算需要消耗极大的算力资源。所以为了有效且高效地利用二阶信息,提出一种新的随机L-BFGS硬阈值优化算法用于解决非凸的稀疏学习问题,算法将迭代硬阈值方法(IHT)引入随机L-BFGS算法,在保持模型性能的同时显著提升了算法的收敛速度,并在线性回归和逻辑回归上的实验结果上证明了新算法的优越性。

First-order optimization methods have been widely applied in the learning of sparse models. The fundamental idea of these methods is to incorporate the Iterative Hard Thresholding(IHT) algorithm into traditional optimization methods. Compared with the first-order methods, very few second-order methods have been applied to sparse optimization problems because of the tremendous computation that obtains the Hessian matrix and its inverse. In order to make use of the second-order information effectively and efficiently, this paper proposes a novel optimization method to solve the nonconvex sparse learning problems. The core idea of the proposed Stochastic L-BFGS Hard Thresholding approach is to incorporate the Stochastic L-BFGS into the Iterative Hard Thresholding(IHT) method, which significantly accelerates the convergence rate and maintains the effectiveness of the model. The experimental results on linear regression and logistic regression demonstrate the superiority of the proposed approach.