We introduce a new algorithm,extended regularized dual averaging(XRDA),for solving regularized stochastic optimization problems,which generalizes the regularized dual averaging(RDA)method.The main novelty of the metho...We introduce a new algorithm,extended regularized dual averaging(XRDA),for solving regularized stochastic optimization problems,which generalizes the regularized dual averaging(RDA)method.The main novelty of the method is that it allows a flexible control of the backward step size.For instance,the backward step size used in RDA grows without bound,while for XRDA the backward step size can be kept bounded.We demonstrate experimentally that additional control over the backward step size can speed up the convergence of the algorithm while preserving desired properties of the iterates,such as sparsity.Theoretically,we show that the XRDA method achieves the same convergence rate as RDA for general convex objectives.展开更多
We develop a generalization of Nesterov’s accelerated gradient descent method which is designed to deal with orthogonality constraints.To demonstrate the effectiveness of our method,we perform numerical experiments w...We develop a generalization of Nesterov’s accelerated gradient descent method which is designed to deal with orthogonality constraints.To demonstrate the effectiveness of our method,we perform numerical experiments which demonstrate that the number of iterations scales with the square root of the condition number,and also compare with existing state-of-the-art quasi-Newton methods on the Stiefel manifold.Our experiments show that our method outperforms existing state-of-the-art quasi-Newton methods on some large,ill-conditioned problems.展开更多
基金Verne M.Willaman Fund,the Center for Computational Mathematics and Applications(CCMA)at the Pennsylvania State University,and the National Science Foundation(Grant No.DMS-1819157).
文摘We introduce a new algorithm,extended regularized dual averaging(XRDA),for solving regularized stochastic optimization problems,which generalizes the regularized dual averaging(RDA)method.The main novelty of the method is that it allows a flexible control of the backward step size.For instance,the backward step size used in RDA grows without bound,while for XRDA the backward step size can be kept bounded.We demonstrate experimentally that additional control over the backward step size can speed up the convergence of the algorithm while preserving desired properties of the iterates,such as sparsity.Theoretically,we show that the XRDA method achieves the same convergence rate as RDA for general convex objectives.
文摘We develop a generalization of Nesterov’s accelerated gradient descent method which is designed to deal with orthogonality constraints.To demonstrate the effectiveness of our method,we perform numerical experiments which demonstrate that the number of iterations scales with the square root of the condition number,and also compare with existing state-of-the-art quasi-Newton methods on the Stiefel manifold.Our experiments show that our method outperforms existing state-of-the-art quasi-Newton methods on some large,ill-conditioned problems.