Stochastic Gradient Compression for Federated Learning over Wireless Network

As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dimensional stochastic gradients to edge server in training,which cause severe communication bottleneck.To address this problem,we compress the communication by sparsifying and quantizing the stochastic gradients of edge devices.We first derive a closed form of the communication compression in terms of sparsification and quantization factors.Then,the convergence rate of this communicationcompressed system is analyzed and several insights are obtained.Finally,we formulate and deal with the quantization resource allocation problem for the goal of minimizing the convergence upper bound,under the constraint of multiple-access channel capacity.Simulations show that the proposed scheme outperforms the benchmarks.

L_(1)-Smooth SVM with Distributed Adaptive Proximal Stochastic Gradient Descent with Momentum for Fast Brain Tumor Detection

Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for gaining a deeper understanding of tumors and improving treatment outcomes.Machine learning models have become key players in automating brain tumor detection.Gradient descent methods are the mainstream algorithms for solving machine learning models.In this paper,we propose a novel distributed proximal stochastic gradient descent approach to solve the L_(1)-Smooth Support Vector Machine(SVM)classifier for brain tumor detection.Firstly,the smooth hinge loss is introduced to be used as the loss function of SVM.It avoids the issue of nondifferentiability at the zero point encountered by the traditional hinge loss function during gradient descent optimization.Secondly,the L_(1) regularization method is employed to sparsify features and enhance the robustness of the model.Finally,adaptive proximal stochastic gradient descent(PGD)with momentum,and distributed adaptive PGDwithmomentum(DPGD)are proposed and applied to the L_(1)-Smooth SVM.Distributed computing is crucial in large-scale data analysis,with its value manifested in extending algorithms to distributed clusters,thus enabling more efficient processing ofmassive amounts of data.The DPGD algorithm leverages Spark,enabling full utilization of the computer’s multi-core resources.Due to its sparsity induced by L_(1) regularization on parameters,it exhibits significantly accelerated convergence speed.From the perspective of loss reduction,DPGD converges faster than PGD.The experimental results show that adaptive PGD withmomentumand its variants have achieved cutting-edge accuracy and efficiency in brain tumor detection.Frompre-trained models,both the PGD andDPGD outperform other models,boasting an accuracy of 95.21%.

Almost Sure Convergence of Proximal Stochastic Accelerated Gradient Methods

Proximal gradient descent and its accelerated version are resultful methods for solving the sum of smooth and non-smooth problems. When the smooth function can be represented as a sum of multiple functions, the stochastic proximal gradient method performs well. However, research on its accelerated version remains unclear. This paper proposes a proximal stochastic accelerated gradient (PSAG) method to address problems involving a combination of smooth and non-smooth components, where the smooth part corresponds to the average of multiple block sums. Simultaneously, most of convergence analyses hold in expectation. To this end, under some mind conditions, we present an almost sure convergence of unbiased gradient estimation in the non-smooth setting. Moreover, we establish that the minimum of the squared gradient mapping norm arbitrarily converges to zero with probability one.

Outage Probability Analysis for D2D-Enabled Heterogeneous Cellular Networks with Exclusion Zone:A Stochastic Geometry Approach

Interference management is one of the most important issues in the device-to-device(D2D)-enabled heterogeneous cellular networks(HetCNets)due to the coexistence of massive cellular and D2D devices in which D2D devices reuse the cellular spectrum.To alleviate the interference,an efficient interference management way is to set exclusion zones around the cellular receivers.In this paper,we adopt a stochastic geometry approach to analyze the outage probabilities of cellular and D2D users in the D2D-enabled HetCNets.The main difficulties contain three aspects:1)how to model the location randomness of base stations,cellular and D2D users in practical networks;2)how to capture the randomness and interrelation of cellular and D2D transmissions due to the existence of random exclusion zones;3)how to characterize the different types of interference and their impacts on the outage probabilities of cellular and D2D users.We then run extensive Monte-Carlo simulations which manifest that our theoretical model is very accurate.

Efficient and High-quality Recommendations via Momentum-incorporated Parallel Stochastic Gradient Descent-Based Learning

A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems.Aiming at addressing this issue,this study proposes a momentum-incorporated parallel stochastic gradient descent(MPSGD)algorithm,whose main idea is two-fold:a)implementing parallelization via a novel datasplitting strategy,and b)accelerating convergence rate by integrating momentum effects into its training process.With it,an MPSGD-based latent factor(MLF)model is achieved,which is capable of performing efficient and high-quality recommendations.Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm,an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.

A New Conjugate Gradient Projection Method for Solving Stochastic Generalized Linear Complementarity Problems

In this paper, a class of the stochastic generalized linear complementarity problems with finitely many elements is proposed for the first time. Based on the Fischer-Burmeister function, a new conjugate gradient projection method is given for solving the stochastic generalized linear complementarity problems. The global convergence of the conjugate gradient projection method is proved and the related numerical results are also reported.

CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

Online gradient algorithm has been widely used as a learning algorithm for feedforward neural network training. In this paper, we prove a weak convergence theorem of an online gradient algorithm with a penalty term, assuming that the training examples are input in a stochastic way. The monotonicity of the error function in the iteration and the boundedness of the weight are both guaranteed. We also present a numerical experiment to support our results.

FL-EASGD:Federated Learning Privacy Security Method Based on Homomorphic Encryption

Federated learning ensures data privacy and security by sharing models among multiple computing nodes instead of plaintext data.However,there is still a potential risk of privacy leakage,for example,attackers can obtain the original data through model inference attacks.Therefore,safeguarding the privacy of model parameters becomes crucial.One proposed solution involves incorporating homomorphic encryption algorithms into the federated learning process.However,the existing federated learning privacy protection scheme based on homomorphic encryption will greatly reduce the efficiency and robustness when there are performance differences between parties or abnormal nodes.To solve the above problems,this paper proposes a privacy protection scheme named Federated Learning-Elastic Averaging Stochastic Gradient Descent(FL-EASGD)based on a fully homomorphic encryption algorithm.First,this paper introduces the homomorphic encryption algorithm into the FL-EASGD scheme to preventmodel plaintext leakage and realize privacy security in the process ofmodel aggregation.Second,this paper designs a robust model aggregation algorithm by adding time variables and constraint coefficients,which ensures the accuracy of model prediction while solving performance differences such as computation speed and node anomalies such as downtime of each participant.In addition,the scheme in this paper preserves the independent exploration of the local model by the nodes of each party,making the model more applicable to the local data distribution.Finally,experimental analysis shows that when there are abnormalities in the participants,the efficiency and accuracy of the whole protocol are not significantly affected.

Differentially private SGD with random features

In the realm of large-scale machine learning,it is crucial to explore methods for reducing computational complexity and memory demands while maintaining generalization performance.Additionally,since the collected data may contain some sensitive information,it is also of great significance to study privacy-preserving machine learning algorithms.This paper focuses on the performance of the differentially private stochastic gradient descent(SGD)algorithm based on random features.To begin,the algorithm maps the original data into a lowdimensional space,thereby avoiding the traditional kernel method for large-scale data storage requirement.Subsequently,the algorithm iteratively optimizes parameters using the stochastic gradient descent approach.Lastly,the output perturbation mechanism is employed to introduce random noise,ensuring algorithmic privacy.We prove that the proposed algorithm satisfies the differential privacy while achieving fast convergence rates under some mild conditions.

Auxiliary Model Based Multi-innovation Stochastic Gradient Identification Methods for Hammerstein Output-Error System

Special input signals identification method based on the auxiliary model based multi-innovation stochastic gradient algorithm for Hammerstein output-error system was proposed.The special input signals were used to realize the identification and separation of the Hammerstein model.As a result,the identification of the dynamic linear part can be separated from the static nonlinear elements without any redundant adjustable parameters.The auxiliary model based multi-innovation stochastic gradient algorithm was applied to identifying the serial link parameters of the Hammerstein model.The auxiliary model based multi-innovation stochastic gradient algorithm can avoid the influence of noise and improve the identification accuracy by changing the innovation length.The simulation results show the efficiency of the proposed method.

Predicted Oil Recovery Scaling-Law Using Stochastic Gradient Boosting Regression Model

In the process of oil recovery,experiments are usually carried out on core samples to evaluate the recovery of oil,so the numerical data are fitted into a non-dimensional equation called scaling-law.This will be essential for determining the behavior of actual reservoirs.The global non-dimensional time-scale is a parameter for predicting a realistic behavior in the oil field from laboratory data.This non-dimensional universal time parameter depends on a set of primary parameters that inherit the properties of the reservoir fluids and rocks and the injection velocity,which dynamics of the process.One of the practical machine learning(ML)techniques for regression/classification problems is gradient boosting(GB)regression.The GB produces a prediction model as an ensemble of weak prediction models that can be done at each iteration by matching a least-squares base-learner with the current pseudoresiduals.Using a randomization process increases the execution speed and accuracy of GB.Hence in this study,we developed a stochastic regression model of gradient boosting(SGB)to forecast oil recovery.Different nondimensional time-scales have been used to generate data to be used with machine learning techniques.The SGB method has been found to be the best machine learning technique for predicting the non-dimensional time-scale,which depends on oil/rock properties.

Stochastic Gradient Boosting Model for Twitter Spam Detection

In today’s world of connectivity there is a huge amount of data than we could imagine.The number of network users are increasing day by day and there are large number of social networks which keeps the users connected all the time.These social networks give the complete independence to the user to post the data either political,commercial or entertainment value.Some data may be sensitive and have a greater impact on the society as a result.The trustworthiness of data is important when it comes to public social networking sites like facebook and twitter.Due to the large user base and its openness there is a huge possibility to spread spam messages in this network.Spam detection is a technique to identify and mark data as a false data value.There are lot of machine learning approaches proposed to detect spam in social networks.The efficiency of any spam detection algorithm is determined by its cost factor and accuracy.Aiming to improve the detection of spam in the social networks this study proposes using statistical based features that are modelled through the supervised boosting approach called Stochastic gradient boosting to evaluate the twitter data sets in the English language.The performance of the proposed model is evaluated using simulation results.

Stochastic Finite Element Method for Mechanical Vibration Based on Conjugate Gradient(CG)

When material properties, geometry parameters and applied loads are assumed to be stochastic, the vibration equation of a system is transformed to static problem by using Newmark method. In order to improve the computational efficiency and to save storage, the Conjugate Gradient （CG） method is presented. The CG is an effective method for solving a large system of linear equations and belongs to the method of iteration with rapid convergence and high precision. An example is given and calculated results are compared to validate the proposed methods.

A stochastic gradient-based two-step sparse identification algorithm for multivariate ARX systems

We consider the sparse identification of multivariate ARX systems, i.e., to recover the zero elements of the unknown parameter matrix. We propose a two-step algorithm, where in the first step the stochastic gradient (SG) algorithm is applied to obtain initial estimates of the unknown parameter matrix and in the second step an optimization criterion is introduced for the sparse identification of multivariate ARX systems. Under mild conditions, we prove that by minimizing the criterion function, the zero elements of the unknown parameter matrix can be recovered with a finite number of observations. The performance of the algorithm is testified through a simulation example.

A Stochastic Approximation Frame Algorithm with Adaptive Directions

Stochastic approximation problem is to find some root or extremum of a non- linear function for which only noisy measurements of the function are available.The classical algorithm for stochastic approximation problem is the Robbins-Monro (RM) algorithm,which uses the noisy evaluation of the negative gradient direction as the iterative direction.In order to accelerate the RM algorithm,this paper gives a flame algorithm using adaptive iterative directions.At each iteration,the new algorithm goes towards either the noisy evaluation of the negative gradient direction or some other directions under some switch criterions.Two feasible choices of the criterions are pro- posed and two corresponding flame algorithms are formed.Different choices of the directions under the same given switch criterion in the flame can also form different algorithms.We also proposed the simultanous perturbation difference forms for the two flame algorithms.The almost surely convergence of the new algorithms are all established.The numerical experiments show that the new algorithms are promising.

基于SGDM优化IWOA-CNN的配电网工程造价控制研究

为了控制配电网工程项目的成本,需准确预测配电网工程造价,本文提出一种基于带动量因子的随机梯度下降(stochastic gradient descent with momentum factor, SGDM)优化的改进鲸鱼算法-卷积神经网络工程造价预测模型。首先,考虑回路数、杆塔数、导线、地形、地质、风速、覆冰、导线截面、混凝土杆、塔材、绝缘子(直线)、绝缘子(耐张)、基坑开方、基础钢材、底盘和水泥对配电网工程造价的影响,建立了非线性函数关系;采用SGDM优化器改进的卷积神经网络对函数进行逼近,并用贝叶斯方法优化卷积神经网络的超参数;利用改进的鲸鱼算法(improved whale optimization algorithm, IWOA)优化卷积神经网络,找出卷积神经网络的最优学习率。数值算例表明,新模型预测效果较好,并提出相应的控制策略。

基于Local SGD的部分同步通信策略

Local SGD训练方法用于分布式机器学习以缓解通信瓶颈,但其本地多轮迭代特性使异构集群节点计算时间差距增大,带来较大同步时延与参数陈旧问题。针对上述问题,基于Local SGD方法提出了一种动态部分同步通信策略(LPSP),该方法利用两层决策充分发挥Local SGD本地迭代优势。在节点每轮迭代计算结束后,基于本地训练情况判断通信可能性,并在全局划分同步集合以最小化同步等待时延,减少Local SGD通信开销并有效控制straggler负面影响。实验表明LPSP可以在不损失训练精确度的情况下实现最高0.75~1.26倍的加速,此外,最高还有5.14%的精确度提升,可以有效加速训练收敛。

基于SGD算法优化的BP神经网络围岩参数反演模型研究

为充分利用现场监测数据所反馈的围岩变形信息,对岩体力学参数进行反演,以贵州省剑河至黎平高速公路TJ-1标段牛练塘隧道为工程背景,选择围岩弹性模量、黏聚力、泊松比及内摩擦角为影响因素,通过设计正交试验及有限元模拟,获取25组围岩物理力学参数组合及其对应的拱顶沉降值和拱腰收敛模拟值。基于随机梯度下降算法(stochastic gradient descent algorithm,简称SGD算法)对传统BP神经网络模型进行改进,建立以拱顶沉降值和拱腰收敛值为输入参数,以围岩弹性模量、黏聚力、泊松比及内摩擦角为输出值的基于SGD算法优化的BP神经网络模型,实现围岩参数的反演分析。将反演所得的围岩参数代入有限元模型,验证优化BP神经网络模型的可行性和准确性。最后,分析围岩变形及初期支护受力特性并给出施工建议。结果表明:1)基于SGD算法优化的BP神经网络模型计算得出的拱顶沉降值、拱腰收敛值、拱肩收敛值与现场实测值的相对误差率在2.50%~24.01%,均低于传统BP神经网络模型计算得出的误差率(11.51%~93.71%),验证优化BP神经网络模型的可行性和优越性;2)上、下台阶拱脚处的喷层和锚杆有应力集中现象,有破坏风险,建议施工中加强拱脚支护,防止发生工程事故。

On complete and micropolar-based incomplete strain gradient theories for periodic lattice structures

The micropolar(MP) and strain gradient(SG) continua have been generally adopted to investigate the relations between the macroscopic elastic constants and the microstructural geometric parameters. Owing to the fact that the microrotation in the MP theory can be expressed in terms of the displacement gradient components, we may regard the MP theory as a particular incomplete SG theory called the MPSG theory,compared with the existing SG theories which are deemed complete since all the SGs are included. Taking the triangular lattice comprising zigzag beams as an example, it is found that as the angle of the zigzag beams increases, the bending of the beams plays a more important role in the total strain energy, and the difference between the results by the two theories gradually decreases. Finally, the models are verified with the pure bending and simple shear of lattices by comparing with the results obtained by the finite element method(FEM)-based structure analyses.