L_(1)-Smooth SVM with Distributed Adaptive Proximal Stochastic Gradient Descent with Momentum for Fast Brain Tumor Detection

Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for gaining a deeper understanding of tumors and improving treatment outcomes.Machine learning models have become key players in automating brain tumor detection.Gradient descent methods are the mainstream algorithms for solving machine learning models.In this paper,we propose a novel distributed proximal stochastic gradient descent approach to solve the L_(1)-Smooth Support Vector Machine(SVM)classifier for brain tumor detection.Firstly,the smooth hinge loss is introduced to be used as the loss function of SVM.It avoids the issue of nondifferentiability at the zero point encountered by the traditional hinge loss function during gradient descent optimization.Secondly,the L_(1) regularization method is employed to sparsify features and enhance the robustness of the model.Finally,adaptive proximal stochastic gradient descent(PGD)with momentum,and distributed adaptive PGDwithmomentum(DPGD)are proposed and applied to the L_(1)-Smooth SVM.Distributed computing is crucial in large-scale data analysis,with its value manifested in extending algorithms to distributed clusters,thus enabling more efficient processing ofmassive amounts of data.The DPGD algorithm leverages Spark,enabling full utilization of the computer’s multi-core resources.Due to its sparsity induced by L_(1) regularization on parameters,it exhibits significantly accelerated convergence speed.From the perspective of loss reduction,DPGD converges faster than PGD.The experimental results show that adaptive PGD withmomentumand its variants have achieved cutting-edge accuracy and efficiency in brain tumor detection.Frompre-trained models,both the PGD andDPGD outperform other models,boasting an accuracy of 95.21%.

Fractional Gradient Descent RBFNN for Active Fault-Tolerant Control of Plant Protection UAVs

With the increasing prevalence of high-order systems in engineering applications, these systems often exhibitsignificant disturbances and can be challenging to model accurately. As a result, the active disturbance rejectioncontroller (ADRC) has been widely applied in various fields. However, in controlling plant protection unmannedaerial vehicles (UAVs), which are typically large and subject to significant disturbances, load disturbances andthe possibility of multiple actuator faults during pesticide spraying pose significant challenges. To address theseissues, this paper proposes a novel fault-tolerant control method that combines a radial basis function neuralnetwork (RBFNN) with a second-order ADRC and leverages a fractional gradient descent (FGD) algorithm.We integrate the plant protection UAV model’s uncertain parameters, load disturbance parameters, and actuatorfault parameters and utilize the RBFNN for system parameter identification. The resulting ADRC exhibits loaddisturbance suppression and fault tolerance capabilities, and our proposed active fault-tolerant control law hasLyapunov stability implications. Experimental results obtained using a multi-rotor fault-tolerant test platformdemonstrate that the proposed method outperforms other control strategies regarding load disturbance suppressionand fault-tolerant performance.

Rockburst Intensity Grade Prediction Model Based on Batch Gradient Descent and Multi-Scale Residual Deep Neural Network

Rockburst is a phenomenon in which free surfaces are formed during excavation,which subsequently causes the sudden release of energy in the construction of mines and tunnels.Light rockburst only peels off rock slices without ejection,while severe rockburst causes casualties and property loss.The frequency and degree of rockburst damage increases with the excavation depth.Moreover,rockburst is the leading engineering geological hazard in the excavation process,and thus the prediction of its intensity grade is of great significance to the development of geotechnical engineering.Therefore,the prediction of rockburst intensity grade is one problem that needs to be solved urgently.By comprehensively considering the occurrence mechanism of rockburst,this paper selects the stress index(σθ/σc),brittleness index(σ_(c)/σ_(t)),and rock elastic energy index(Wet)as the rockburst evaluation indexes through the Spearman coefficient method.This overcomes the low accuracy problem of a single evaluation index prediction method.Following this,the BGD-MSR-DNN rockburst intensity grade prediction model based on batch gradient descent and a multi-scale residual deep neural network is proposed.The batch gradient descent(BGD)module is used to replace the gradient descent algorithm,which effectively improves the efficiency of the network and reduces the model training time.Moreover,the multi-scale residual(MSR)module solves the problem of network degradation when there are too many hidden layers of the deep neural network(DNN),thus improving the model prediction accuracy.The experimental results reveal the BGDMSR-DNN model accuracy to reach 97.1%,outperforming other comparable models.Finally,actual projects such as Qinling Tunnel and Daxiangling Tunnel,reached an accuracy of 100%.The model can be applied in mines and tunnel engineering to realize the accurate and rapid prediction of rockburst intensity grade.

Stochastic Gradient Compression for Federated Learning over Wireless Network

As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dimensional stochastic gradients to edge server in training,which cause severe communication bottleneck.To address this problem,we compress the communication by sparsifying and quantizing the stochastic gradients of edge devices.We first derive a closed form of the communication compression in terms of sparsification and quantization factors.Then,the convergence rate of this communicationcompressed system is analyzed and several insights are obtained.Finally,we formulate and deal with the quantization resource allocation problem for the goal of minimizing the convergence upper bound,under the constraint of multiple-access channel capacity.Simulations show that the proposed scheme outperforms the benchmarks.

Differentially private SGD with random features

In the realm of large-scale machine learning,it is crucial to explore methods for reducing computational complexity and memory demands while maintaining generalization performance.Additionally,since the collected data may contain some sensitive information,it is also of great significance to study privacy-preserving machine learning algorithms.This paper focuses on the performance of the differentially private stochastic gradient descent(SGD)algorithm based on random features.To begin,the algorithm maps the original data into a lowdimensional space,thereby avoiding the traditional kernel method for large-scale data storage requirement.Subsequently,the algorithm iteratively optimizes parameters using the stochastic gradient descent approach.Lastly,the output perturbation mechanism is employed to introduce random noise,ensuring algorithmic privacy.We prove that the proposed algorithm satisfies the differential privacy while achieving fast convergence rates under some mild conditions.

GDLIN:一种利用梯度下降的学习索引

在大数据时代,数据访问速度是衡量大规模存储系统性能的一个重要指标,而索引是用于提升数据库系统中数据存取性能的主要技术之一。近几年,使用机器学习模型代替B+树等传统索引,拟合数据分布规律,将数据的间接查找优化为函数直接计算的学习索引(Learned Index,LI)被提出,LI提高了查询的速度,减少了索引空间开销。但是LI的拟合误差较大,不支持插入等修改性操作。文中提出了一种利用梯度下降算法拟合数据的学习索引模型GDLIN(A Learned Index By Gradient Descent)。GDLIN利用梯度下降算法更好地拟合数据,减少拟合误差,缩短本地查找的时间;同时递归调用数据拟合算法,充分利用键的分布规律,构建上层结构,避免索引结构随着数据量而增大。另外,GDLIN利用链表解决LI不支持数据插入的问题。实验结果表明,GDLIN在无新数据插入的情况下,吞吐量是B+树的2.1倍;在插入操作占比为50%的情况下,是LI的1.08倍。

Efficient and High-quality Recommendations via Momentum-incorporated Parallel Stochastic Gradient Descent-Based Learning

A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems.Aiming at addressing this issue,this study proposes a momentum-incorporated parallel stochastic gradient descent(MPSGD)algorithm,whose main idea is two-fold:a)implementing parallelization via a novel datasplitting strategy,and b)accelerating convergence rate by integrating momentum effects into its training process.With it,an MPSGD-based latent factor(MLF)model is achieved,which is capable of performing efficient and high-quality recommendations.Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm,an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.

A modified three–term conjugate gradient method with sufficient descent property

A hybridization of the three–term conjugate gradient method proposed by Zhang et al. and the nonlinear conjugate gradient method proposed by Polak and Ribi`ere, and Polyak is suggested. Based on an eigenvalue analysis, it is shown that search directions of the proposed method satisfy the sufficient descent condition, independent of the line search and the objective function convexity. Global convergence of the method is established under an Armijo–type line search condition. Numerical experiments show practical efficiency of the proposed method.

基于SGDM优化IWOA-CNN的配电网工程造价控制研究

为了控制配电网工程项目的成本,需准确预测配电网工程造价,本文提出一种基于带动量因子的随机梯度下降(stochastic gradient descent with momentum factor, SGDM)优化的改进鲸鱼算法-卷积神经网络工程造价预测模型。首先,考虑回路数、杆塔数、导线、地形、地质、风速、覆冰、导线截面、混凝土杆、塔材、绝缘子(直线)、绝缘子(耐张)、基坑开方、基础钢材、底盘和水泥对配电网工程造价的影响,建立了非线性函数关系;采用SGDM优化器改进的卷积神经网络对函数进行逼近,并用贝叶斯方法优化卷积神经网络的超参数;利用改进的鲸鱼算法(improved whale optimization algorithm, IWOA)优化卷积神经网络,找出卷积神经网络的最优学习率。数值算例表明,新模型预测效果较好,并提出相应的控制策略。

Designing fuzzy inference system based on improved gradient descent method

The distribution of sampling data influences completeness of rule base so that extrapolating missing rules is very difficult. Based on data mining, a self-learning method is developed for identifying fuzzy model and extrapolating missing rules, by means of confidence measure and the improved gradient descent method. The proposed approach can not only identify fuzzy model, update its parameters and determine optimal output fuzzy sets simultaneously, but also resolve the uncontrollable problem led by the regions that data do not cover. The simulation results show the effectiveness and accuracy of the proposed approach with the classical truck backer-upper control problem verifying.

Gradient Descent Algorithm for Small UAV Parameter Estimation System

A gradient descent algorithm with adjustable parameter for attitude estimation is developed,aiming at the attitude measurement for small unmanned aerial vehicle(UAV)in real-time flight conditions.The accelerometer and magnetometer are introduced to construct an error equation with the gyros,thus the drifting characteristics of gyroscope can be compensated by solving the error equation utilized by the gradient descent algorithm.Performance of the presented algorithm is evaluated using a self-proposed micro-electro-mechanical system(MEMS)based attitude heading reference system which is mounted on a tri-axis turntable.The on-ground,turntable and flight experiments indicate that the estimation attitude has a good accuracy.Also,the presented system is compared with an open-source flight control system which runs extended Kalman filter(EKF),and the results show that the attitude control system using the gradient descent method can estimate the attitudes for UAV effectively.

PROJECTED GRADIENT DESCENT BASED ON SOFT THRESHOLDING IN MATRIX COMPLETION

Matrix completion is the extension of compressed sensing.In compressed sensing,we solve the underdetermined equations using sparsity prior of the unknown signals.However,in matrix completion,we solve the underdetermined equations based on sparsity prior in singular values set of the unknown matrix,which also calls low-rank prior of the unknown matrix.This paper firstly introduces basic concept of matrix completion,analyses the matrix suitably used in matrix completion,and shows that such matrix should satisfy two conditions:low rank and incoherence property.Then the paper provides three reconstruction algorithms commonly used in matrix completion:singular value thresholding algorithm,singular value projection,and atomic decomposition for minimum rank approximation,puts forward their shortcoming to know the rank of original matrix.The Projected Gradient Descent based on Soft Thresholding(STPGD),proposed in this paper predicts the rank of unknown matrix using soft thresholding,and iteratives based on projected gradient descent,thus it could estimate the rank of unknown matrix exactly with low computational complexity,this is verified by numerical experiments.We also analyze the convergence and computational complexity of the STPGD algorithm,point out this algorithm is guaranteed to converge,and analyse the number of iterations needed to reach reconstruction error.Compared the computational complexity of the STPGD algorithm to other algorithms,we draw the conclusion that the STPGD algorithm not only reduces the computational complexity,but also improves the precision of the reconstruction solution.

A Descent Gradient Method and Its Global Convergence

Y Liu and C Storey(1992)proposed the famous LS conjugate gradient method which has good numerical results.However,the LS method has very weak convergence under the Wolfe-type line search.In this paper,we give a new descent gradient method based on the LS method.It can guarantee the sufficient descent property at each iteration and the global convergence under the strong Wolfe line search.Finally,we also present extensive preliminary numerical experiments to show the efficiency of the proposed method by comparing with the famous PRP^+method.

Linear Regression and Gradient Descent Method for Electricity Output Power Prediction

Regulating the power output for a power plant as demand for electricity fluctuates throughout the day is important for both economic purpose and the safety of the generator. In this work, gradient descent method together with regularization is investigated to study the electricity output related to vacuum level and temperature in the turbine. Ninety percent of the data was used to train the regression parameters while the remaining ten percent was used for validation. Final results showed that 99% accuracy could be obtained with this method. This opens a new window for electricity output prediction for power plants.

A New Descent Nonlinear Conjugate Gradient Method for Unconstrained Optimization

In this paper, a new nonlinear conjugate gradient method is proposed for large-scale unconstrained optimization. The sufficient descent property holds without any line searches. We use some steplength technique which ensures the Zoutendijk condition to be held, this method is proved to be globally convergent. Finally, we improve it, and do further analysis.

基于ILSTM-AMSGD神经网络的时间序列预测方法

针对标准长短期记忆(long short-term memory,LSTM)神经网络的结构参数众多、训练过程耗时长问题,提出一种基于自适应动量随机梯度下降(adaptive momentum stochastic gradient descent,AMSGD)算法的改进型长短期记忆神经网络(ILSTM-AMSGD神经网络),并将其用于时间序列预测中。首先,通过简化结构方程中的递归项权值,减少网络中所需训练的参数。其次,设计一种AMSGD算法对神经网络结构参数进行学习。最后,通过2个基准数据集和1个实际数据集对ILSTM-AMSGD神经网络模型在时间序列预测中的准确性和运行效率进行实验验证。结果表明,递归项权值简化方法可以提高模型的泛化能力,同时AMSGD算法加快了模型的收敛速度。与其他模型相比,ILSTM-AMSGD神经网络模型实现了对时间序列更加高效、准确的预测。

基于Local SGD的部分同步通信策略

Local SGD训练方法用于分布式机器学习以缓解通信瓶颈,但其本地多轮迭代特性使异构集群节点计算时间差距增大,带来较大同步时延与参数陈旧问题。针对上述问题,基于Local SGD方法提出了一种动态部分同步通信策略(LPSP),该方法利用两层决策充分发挥Local SGD本地迭代优势。在节点每轮迭代计算结束后,基于本地训练情况判断通信可能性,并在全局划分同步集合以最小化同步等待时延,减少Local SGD通信开销并有效控制straggler负面影响。实验表明LPSP可以在不损失训练精确度的情况下实现最高0.75~1.26倍的加速,此外,最高还有5.14%的精确度提升,可以有效加速训练收敛。

EFFICIENT GRADIENT DESCENT METHOD OFRBF NEURAL ENTWORKS WITHADAPTIVE LEARNING RATE

A new algorithm to exploit the learning rates of gradient descent method is presented, based on the second-order Taylor expansion of the error energy function with respect to learning rate, at some values decided by "award-punish" strategy. Detailed deduction of the algorithm applied to RBF networks is given. Simulation studies show that this algorithm can increase the rate of convergence and improve the performance of the gradient descent method.

An Efficient Energy Routing Protocol Based on Gradient Descent Method in WSNs

In a wireless sensor network[1],the operation of a node depends on the battery power it carries.Because of the environmental reasons,the node cannot replace the battery.In order to improve the life cycle of the network,energy becomes one of the key problems in the design of the wireless sensor network(WSN)routing protocol[2].This paper proposes a routing protocol ERGD based on the method of gradient descent that can minimizes the consumption of energy.Within the communication radius of the current node,the distance between the current node and the next hop node is assumed that can generate a projected energy at the distance from the current node to the base station(BS),this projected energy and the remaining energy of the next hop node is the key factor in finding the next hop node.The simulation results show that the proposed protocol effectively extends the life cycle of the network and improves the reliability and fault tolerance of the system.

基于SGD算法优化的BP神经网络围岩参数反演模型研究

为充分利用现场监测数据所反馈的围岩变形信息,对岩体力学参数进行反演,以贵州省剑河至黎平高速公路TJ-1标段牛练塘隧道为工程背景,选择围岩弹性模量、黏聚力、泊松比及内摩擦角为影响因素,通过设计正交试验及有限元模拟,获取25组围岩物理力学参数组合及其对应的拱顶沉降值和拱腰收敛模拟值。基于随机梯度下降算法(stochastic gradient descent algorithm,简称SGD算法)对传统BP神经网络模型进行改进,建立以拱顶沉降值和拱腰收敛值为输入参数,以围岩弹性模量、黏聚力、泊松比及内摩擦角为输出值的基于SGD算法优化的BP神经网络模型,实现围岩参数的反演分析。将反演所得的围岩参数代入有限元模型,验证优化BP神经网络模型的可行性和准确性。最后,分析围岩变形及初期支护受力特性并给出施工建议。结果表明:1)基于SGD算法优化的BP神经网络模型计算得出的拱顶沉降值、拱腰收敛值、拱肩收敛值与现场实测值的相对误差率在2.50%~24.01%,均低于传统BP神经网络模型计算得出的误差率(11.51%~93.71%),验证优化BP神经网络模型的可行性和优越性;2)上、下台阶拱脚处的喷层和锚杆有应力集中现象,有破坏风险,建议施工中加强拱脚支护,防止发生工程事故。