CONVERGENCE OF BACKPRIOPAG ATION WITH MOMENTUM FOR NETWORK A RCHITECTURES WITH SKIP CONNECTIONS
摘要
We study a class of deep neural networks with architectures that form a directed acyclic graph(DAG).For backpropagation defined by gradient descent with adaptive momentum,we show weights converge for a large class of nonlinear activation functions.'The proof generalizes the results of Wu et al.(2008)who showed convergence for a feed-forward network with one hidden layer.For an example of the effectiveness of DAG architectures,we describe an example of compression through an AutoEncoder,and compare against sequential feed-forward networks under several metrics.
二级参考文献15
-
1D.E. Rumelhart, G.E. Hinton, and R.J. Williams, Learning representations by back-propagating errors, Nature, 323 (1986), 533-536.
-
2M. Torii and M.T. Hagan, Stability of steepest descent with momentum for quadratic functions, IEEE T. Neural Networ., 13:3 (2002), 752-756.
-
3W. Wu, H.M. Shao and D. Qu, Strong convergence for gradient methods for BP networks training, Proceedings of 2005 International Conference on Neural Networks and Brains (ICNN-B'05), Edited by M.-S. Zhao and Z.-C. Shi, Beijing, China, 2005, IEEE Press. pp. 332-334.
-
4Y.X. Yuan, W.Y. Sun, Optimization Theory and Methods, Science Press, Beijing, 2001.
-
5N.M. Zhang, W. Wu, and G.F. Zheng, Convergence of gradient method with momentum for two-layer feedforward neural networks, IEEE T. Neural Networ., 17:2 (2006), 522-525.
-
6N. Ampazis and S.J. Perantonis, Two highly efficient second-order algorithms for training feedfor- ward networds, IEEE T. Neural Networ., 13:5 (2002), 1064-1074.
-
7A. Bhaya and E. Kaszkurewicz, Steepest descent with momentum for quadratic functlons is a version of the conjugate gradient method, Neural Networks, 17 (2004), 65-71.
-
8S.E. Fahlman, Faster learning variations on back propogation: AN empirical study, in Proc. 1933, Connectionist Models Summer School, San Mateo, CA: Morgan Kaufmann, 38-51.
-
9T.L. Fine and S. Mukherjee, Parameter convergence and learning curves for neural networks, Neural Comput., 11 (1999), 747-769.
-
10W. Finnoff, Diffusion approximations for the constant learning rate backpropagation algorithm and resistance to locol minima, Neural Comput., 6 (1994), 285-295.
共引文献4
-
1艾剑良,杨曦中.一种基于自适应神经网络的航空发动机故障诊断方法[J].中国科学：技术科学,2018,48(3):326-335. 被引量：15
-
2张迅.Sigma-Pi-Sigma神经网络的带动量项梯度算法的收敛性[J].温州大学学报（自然科学版）,2018,39(2):17-21. 被引量：1
-
3ZHANG Naimin,ZHANG Ting.Recurrent Neural Networks for Computing the Moore-Penrose Inverse with Momentum Learning[J].Chinese Journal of Electronics,2020,29(6):1039-1045. 被引量：1
-
4彭先伦,谢纲.带动量项的梯度下降算法的收敛性[J].华东理工大学学报（自然科学版）,2021,47(6):779-786. 被引量：2
-
1Kevin K. H. Cheung,Patrick Girardet.Improved Approximation of Layout Problems on Random Graphs[J].Open Journal of Discrete Mathematics,2020,10(1):13-30.
-
2Veronica K.H.Chan,Christine W.Chan.Towards explicit representation of an artificial neural network model: Comparison of two artificial neural network rule extraction approaches[J].Petroleum,2020,6(4):329-339.
-
3Veronica Chan,Christine Chan.Learning from a carbon dioxide capture system dataset: Application of the piecewise neural network algorithm[J].Petroleum,2017,3(1):56-67. 被引量：3
-
4MIN Mengcan,CHEN Xiaofang,XIE Yongfang.Constrained voting extreme learning machine and its application[J].Journal of Systems Engineering and Electronics,2021,32(1):209-219. 被引量：5
