在传感器网络定位问题中,利用接收信号强度RSSI(Received Signal Strength Indication)的定位方法存在着接收信号传播不稳定,定位精度较低的问题。为解决该问题,提出了一种基于阈值Nesterov加速梯度下降NAGT(Nesterov Accelerated Gradi...在传感器网络定位问题中,利用接收信号强度RSSI(Received Signal Strength Indication)的定位方法存在着接收信号传播不稳定,定位精度较低的问题。为解决该问题,提出了一种基于阈值Nesterov加速梯度下降NAGT(Nesterov Accelerated Gradient Descent with Threshold)的RSSI定位算法。算法引入Nesterov思想,不断更新寻优动量,以达到损失函数最小,从而求取对应的未知基站坐标,通过增设阈值,降低了算法陷入局部最优的概率。经仿真比较分析,NAGT方法相对于粒子群算法与随机梯度法,在定位精度与效率上有着较为明显的优势。展开更多
Gradient descent(GD)algorithm is the widely used optimisation method in training machine learning and deep learning models.In this paper,based on GD,Polyak’s momentum(PM),and Nesterov accelerated gradient(NAG),we giv...Gradient descent(GD)algorithm is the widely used optimisation method in training machine learning and deep learning models.In this paper,based on GD,Polyak’s momentum(PM),and Nesterov accelerated gradient(NAG),we give the convergence of the algorithms from an ini-tial value to the optimal value of an objective function in simple quadratic form.Based on the convergence property of the quadratic function,two sister sequences of NAG’s iteration and par-allel tangent methods in neural networks,the three-step accelerated gradient(TAG)algorithm is proposed,which has three sequences other than two sister sequences.To illustrate the perfor-mance of this algorithm,we compare the proposed algorithm with the three other algorithms in quadratic function,high-dimensional quadratic functions,and nonquadratic function.Then we consider to combine the TAG algorithm to the backpropagation algorithm and the stochastic gradient descent algorithm in deep learning.For conveniently facilitate the proposed algorithms,we rewite the R package‘neuralnet’and extend it to‘supneuralnet’.All kinds of deep learning algorithms in this paper are included in‘supneuralnet’package.Finally,we show our algorithms are superior to other algorithms in four case studies.展开更多
This paper is concerned with convergence of stochastic gradient algorithms with momentum terms in the nonconvex setting.A class of stochastic momentum methods,including stochastic gradient descent,heavy ball and Neste...This paper is concerned with convergence of stochastic gradient algorithms with momentum terms in the nonconvex setting.A class of stochastic momentum methods,including stochastic gradient descent,heavy ball and Nesterov’s accelerated gradient,is analyzed in a general framework under mild assumptions.Based on the convergence result of expected gradients,the authors prove the almost sure convergence by a detailed discussion of the effects of momentum and the number of upcrossings.It is worth noting that there are not additional restrictions imposed on the objective function and stepsize.Another improvement over previous results is that the existing Lipschitz condition of the gradient is relaxed into the condition of H?lder continuity.As a byproduct,the authors apply a localization procedure to extend the results to stochastic stepsizes.展开更多
We develop a generalization of Nesterov’s accelerated gradient descent method which is designed to deal with orthogonality constraints.To demonstrate the effectiveness of our method,we perform numerical experiments w...We develop a generalization of Nesterov’s accelerated gradient descent method which is designed to deal with orthogonality constraints.To demonstrate the effectiveness of our method,we perform numerical experiments which demonstrate that the number of iterations scales with the square root of the condition number,and also compare with existing state-of-the-art quasi-Newton methods on the Stiefel manifold.Our experiments show that our method outperforms existing state-of-the-art quasi-Newton methods on some large,ill-conditioned problems.展开更多
基金Supported by the Science and Technology Innovation 2030 New Generation Artificial Intelligence Major Project(2018AAA0100902)the National Key Research and Development Program of China(2019YFB1705800)the National Natural Science Foundation of China(61973270)。
文摘在传感器网络定位问题中,利用接收信号强度RSSI(Received Signal Strength Indication)的定位方法存在着接收信号传播不稳定,定位精度较低的问题。为解决该问题,提出了一种基于阈值Nesterov加速梯度下降NAGT(Nesterov Accelerated Gradient Descent with Threshold)的RSSI定位算法。算法引入Nesterov思想,不断更新寻优动量,以达到损失函数最小,从而求取对应的未知基站坐标,通过增设阈值,降低了算法陷入局部最优的概率。经仿真比较分析,NAGT方法相对于粒子群算法与随机梯度法,在定位精度与效率上有着较为明显的优势。
基金This work was supported by National Natural Science Foun-dation of China(11271136,81530086)Program of Shanghai Subject Chief Scientist(14XD1401600)the 111 Project of China(No.B14019).
文摘Gradient descent(GD)algorithm is the widely used optimisation method in training machine learning and deep learning models.In this paper,based on GD,Polyak’s momentum(PM),and Nesterov accelerated gradient(NAG),we give the convergence of the algorithms from an ini-tial value to the optimal value of an objective function in simple quadratic form.Based on the convergence property of the quadratic function,two sister sequences of NAG’s iteration and par-allel tangent methods in neural networks,the three-step accelerated gradient(TAG)algorithm is proposed,which has three sequences other than two sister sequences.To illustrate the perfor-mance of this algorithm,we compare the proposed algorithm with the three other algorithms in quadratic function,high-dimensional quadratic functions,and nonquadratic function.Then we consider to combine the TAG algorithm to the backpropagation algorithm and the stochastic gradient descent algorithm in deep learning.For conveniently facilitate the proposed algorithms,we rewite the R package‘neuralnet’and extend it to‘supneuralnet’.All kinds of deep learning algorithms in this paper are included in‘supneuralnet’package.Finally,we show our algorithms are superior to other algorithms in four case studies.
基金supported by the National Natural Science Foundation of China (Nos. 11631004,12031009)the National Key R&D Program of China (No. 2018YFA0703900)。
文摘This paper is concerned with convergence of stochastic gradient algorithms with momentum terms in the nonconvex setting.A class of stochastic momentum methods,including stochastic gradient descent,heavy ball and Nesterov’s accelerated gradient,is analyzed in a general framework under mild assumptions.Based on the convergence result of expected gradients,the authors prove the almost sure convergence by a detailed discussion of the effects of momentum and the number of upcrossings.It is worth noting that there are not additional restrictions imposed on the objective function and stepsize.Another improvement over previous results is that the existing Lipschitz condition of the gradient is relaxed into the condition of H?lder continuity.As a byproduct,the authors apply a localization procedure to extend the results to stochastic stepsizes.
文摘We develop a generalization of Nesterov’s accelerated gradient descent method which is designed to deal with orthogonality constraints.To demonstrate the effectiveness of our method,we perform numerical experiments which demonstrate that the number of iterations scales with the square root of the condition number,and also compare with existing state-of-the-art quasi-Newton methods on the Stiefel manifold.Our experiments show that our method outperforms existing state-of-the-art quasi-Newton methods on some large,ill-conditioned problems.
