Gradient descent(GD)algorithm is the widely used optimisation method in training machine learning and deep learning models.In this paper,based on GD,Polyak’s momentum(PM),and Nesterov accelerated gradient(NAG),we giv...Gradient descent(GD)algorithm is the widely used optimisation method in training machine learning and deep learning models.In this paper,based on GD,Polyak’s momentum(PM),and Nesterov accelerated gradient(NAG),we give the convergence of the algorithms from an ini-tial value to the optimal value of an objective function in simple quadratic form.Based on the convergence property of the quadratic function,two sister sequences of NAG’s iteration and par-allel tangent methods in neural networks,the three-step accelerated gradient(TAG)algorithm is proposed,which has three sequences other than two sister sequences.To illustrate the perfor-mance of this algorithm,we compare the proposed algorithm with the three other algorithms in quadratic function,high-dimensional quadratic functions,and nonquadratic function.Then we consider to combine the TAG algorithm to the backpropagation algorithm and the stochastic gradient descent algorithm in deep learning.For conveniently facilitate the proposed algorithms,we rewite the R package‘neuralnet’and extend it to‘supneuralnet’.All kinds of deep learning algorithms in this paper are included in‘supneuralnet’package.Finally,we show our algorithms are superior to other algorithms in four case studies.展开更多
机械钻速(rate of penetration,ROP)是钻井作业优化和减少成本的关键因素,钻井时有效地预测ROP是提升钻进效率的关键。由于井下钻进时复杂多变的情况和地层的非均质性,通过传统的ROP方程和回归分析方法来预测钻速受到了一定的限制。为...机械钻速(rate of penetration,ROP)是钻井作业优化和减少成本的关键因素,钻井时有效地预测ROP是提升钻进效率的关键。由于井下钻进时复杂多变的情况和地层的非均质性,通过传统的ROP方程和回归分析方法来预测钻速受到了一定的限制。为了实现对钻速的高精度预测,对现有BP (back propagation)神经网络进行优化,提出了一种新的神经网络模型,即动态自适应学习率的粒子群优化BP神经网络,利用录井数据建立目标井预测模型来对钻速进行预测。在训练过程中对BP神经网络进行优化,利用启发式算法,即附加动量法和自适应学习率,将两种方法结合起来形成动态自适应学习率的BP改进算法,提高了BP神经网络的训练速度和拟合精度,获得了更好的泛化性能。将BP神经网络与遗传优化算法(genetic algorithm,GA)和粒子群优化算法(particle swarm optimization,PSO)结合,得到优化后的动态自适应学习率BP神经网络。研究利用XX8-1-2井的录井数据进行实验,对比BP神经网络、PSO-BP神经网络、GA-BP神经网络3种不同的改进后神经网络的预测结果。实验结果表明:优化后的PSO-BP神经网络的预测性能最好,具有更高的效率和可靠性,能够有效的利用工程数据,在有一定数据采集量的区域提供较为准确的ROP预测。展开更多
A multilayer perceptron neural network system is established to support the diagnosis for five most common heart diseases (coronary heart disease, rheumatic valvular heart disease, hypertension, chronic cor pulmonale ...A multilayer perceptron neural network system is established to support the diagnosis for five most common heart diseases (coronary heart disease, rheumatic valvular heart disease, hypertension, chronic cor pulmonale and congenital heart disease). Momentum term, adaptive learning rate, the forgetting mechanics, and conjugate gradients method are introduced to improve the basic BP algorithm aiming to speed up the convergence of the BP algorithm and enhance the accuracy for diagnosis. A heart disease database consisting of 352 samples is applied to the training and testing courses of the system. The performance of the system is assessed by cross-validation method. It is found that as the basic BP algorithm is improved step by step, the convergence speed and the classification accuracy of the network are enhanced, and the system has great application prospect in supporting heart diseases diagnosis.展开更多
基金This work was supported by National Natural Science Foun-dation of China(11271136,81530086)Program of Shanghai Subject Chief Scientist(14XD1401600)the 111 Project of China(No.B14019).
文摘Gradient descent(GD)algorithm is the widely used optimisation method in training machine learning and deep learning models.In this paper,based on GD,Polyak’s momentum(PM),and Nesterov accelerated gradient(NAG),we give the convergence of the algorithms from an ini-tial value to the optimal value of an objective function in simple quadratic form.Based on the convergence property of the quadratic function,two sister sequences of NAG’s iteration and par-allel tangent methods in neural networks,the three-step accelerated gradient(TAG)algorithm is proposed,which has three sequences other than two sister sequences.To illustrate the perfor-mance of this algorithm,we compare the proposed algorithm with the three other algorithms in quadratic function,high-dimensional quadratic functions,and nonquadratic function.Then we consider to combine the TAG algorithm to the backpropagation algorithm and the stochastic gradient descent algorithm in deep learning.For conveniently facilitate the proposed algorithms,we rewite the R package‘neuralnet’and extend it to‘supneuralnet’.All kinds of deep learning algorithms in this paper are included in‘supneuralnet’package.Finally,we show our algorithms are superior to other algorithms in four case studies.
基金the Natural Science Foundation of China (No. 30070211).
文摘A multilayer perceptron neural network system is established to support the diagnosis for five most common heart diseases (coronary heart disease, rheumatic valvular heart disease, hypertension, chronic cor pulmonale and congenital heart disease). Momentum term, adaptive learning rate, the forgetting mechanics, and conjugate gradients method are introduced to improve the basic BP algorithm aiming to speed up the convergence of the BP algorithm and enhance the accuracy for diagnosis. A heart disease database consisting of 352 samples is applied to the training and testing courses of the system. The performance of the system is assessed by cross-validation method. It is found that as the basic BP algorithm is improved step by step, the convergence speed and the classification accuracy of the network are enhanced, and the system has great application prospect in supporting heart diseases diagnosis.
