L_(1)-Smooth SVM with Distributed Adaptive Proximal Stochastic Gradient Descent with Momentum for Fast Brain Tumor Detection

Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for gaining a deeper understanding of tumors and improving treatment outcomes.Machine learning models have become key players in automating brain tumor detection.Gradient descent methods are the mainstream algorithms for solving machine learning models.In this paper,we propose a novel distributed proximal stochastic gradient descent approach to solve the L_(1)-Smooth Support Vector Machine(SVM)classifier for brain tumor detection.Firstly,the smooth hinge loss is introduced to be used as the loss function of SVM.It avoids the issue of nondifferentiability at the zero point encountered by the traditional hinge loss function during gradient descent optimization.Secondly,the L_(1) regularization method is employed to sparsify features and enhance the robustness of the model.Finally,adaptive proximal stochastic gradient descent(PGD)with momentum,and distributed adaptive PGDwithmomentum(DPGD)are proposed and applied to the L_(1)-Smooth SVM.Distributed computing is crucial in large-scale data analysis,with its value manifested in extending algorithms to distributed clusters,thus enabling more efficient processing ofmassive amounts of data.The DPGD algorithm leverages Spark,enabling full utilization of the computer’s multi-core resources.Due to its sparsity induced by L_(1) regularization on parameters,it exhibits significantly accelerated convergence speed.From the perspective of loss reduction,DPGD converges faster than PGD.The experimental results show that adaptive PGD withmomentumand its variants have achieved cutting-edge accuracy and efficiency in brain tumor detection.Frompre-trained models,both the PGD andDPGD outperform other models,boasting an accuracy of 95.21%.

Stochastic Gradient Compression for Federated Learning over Wireless Network

As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dimensional stochastic gradients to edge server in training,which cause severe communication bottleneck.To address this problem,we compress the communication by sparsifying and quantizing the stochastic gradients of edge devices.We first derive a closed form of the communication compression in terms of sparsification and quantization factors.Then,the convergence rate of this communicationcompressed system is analyzed and several insights are obtained.Finally,we formulate and deal with the quantization resource allocation problem for the goal of minimizing the convergence upper bound,under the constraint of multiple-access channel capacity.Simulations show that the proposed scheme outperforms the benchmarks.

Efficient and High-quality Recommendations via Momentum-incorporated Parallel Stochastic Gradient Descent-Based Learning

A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems.Aiming at addressing this issue,this study proposes a momentum-incorporated parallel stochastic gradient descent(MPSGD)algorithm,whose main idea is two-fold:a)implementing parallelization via a novel datasplitting strategy,and b)accelerating convergence rate by integrating momentum effects into its training process.With it,an MPSGD-based latent factor(MLF)model is achieved,which is capable of performing efficient and high-quality recommendations.Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm,an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.

求解一类非光滑凸优化问题的相对加速SGD算法

一阶优化算法由于其计算简单、代价小,被广泛应用于机器学习、大数据科学、计算机视觉等领域,然而,现有的一阶算法大多要求目标函数具有Lipschitz连续梯度,而实际中的很多应用问题不满足该要求。在经典的梯度下降算法基础上,引入随机和加速,提出一种相对加速随机梯度下降算法。该算法不要求目标函数具有Lipschitz连续梯度,而是通过将欧氏距离推广为Bregman距离,从而将Lipschitz连续梯度条件减弱为相对光滑性条件。相对加速随机梯度下降算法的收敛性与一致三角尺度指数有关,为避免调节最优一致三角尺度指数参数的工作量,给出一种自适应相对加速随机梯度下降算法。该算法可自适应地选取一致三角尺度指数参数。对算法收敛性的理论分析表明,算法迭代序列的目标函数值收敛于最优目标函数值。针对Possion反问题和目标函数的Hessian阵算子范数随变量范数多项式增长的极小化问题的数值实验表明,自适应相对加速随机梯度下降算法和相对加速随机梯度下降算法的收敛性能优于相对随机梯度下降算法。

Differentially private SGD with random features

In the realm of large-scale machine learning,it is crucial to explore methods for reducing computational complexity and memory demands while maintaining generalization performance.Additionally,since the collected data may contain some sensitive information,it is also of great significance to study privacy-preserving machine learning algorithms.This paper focuses on the performance of the differentially private stochastic gradient descent(SGD)algorithm based on random features.To begin,the algorithm maps the original data into a lowdimensional space,thereby avoiding the traditional kernel method for large-scale data storage requirement.Subsequently,the algorithm iteratively optimizes parameters using the stochastic gradient descent approach.Lastly,the output perturbation mechanism is employed to introduce random noise,ensuring algorithmic privacy.We prove that the proposed algorithm satisfies the differential privacy while achieving fast convergence rates under some mild conditions.

FL-EASGD:Federated Learning Privacy Security Method Based on Homomorphic Encryption

Federated learning ensures data privacy and security by sharing models among multiple computing nodes instead of plaintext data.However,there is still a potential risk of privacy leakage,for example,attackers can obtain the original data through model inference attacks.Therefore,safeguarding the privacy of model parameters becomes crucial.One proposed solution involves incorporating homomorphic encryption algorithms into the federated learning process.However,the existing federated learning privacy protection scheme based on homomorphic encryption will greatly reduce the efficiency and robustness when there are performance differences between parties or abnormal nodes.To solve the above problems,this paper proposes a privacy protection scheme named Federated Learning-Elastic Averaging Stochastic Gradient Descent(FL-EASGD)based on a fully homomorphic encryption algorithm.First,this paper introduces the homomorphic encryption algorithm into the FL-EASGD scheme to preventmodel plaintext leakage and realize privacy security in the process ofmodel aggregation.Second,this paper designs a robust model aggregation algorithm by adding time variables and constraint coefficients,which ensures the accuracy of model prediction while solving performance differences such as computation speed and node anomalies such as downtime of each participant.In addition,the scheme in this paper preserves the independent exploration of the local model by the nodes of each party,making the model more applicable to the local data distribution.Finally,experimental analysis shows that when there are abnormalities in the participants,the efficiency and accuracy of the whole protocol are not significantly affected.

PMUS-HOSGD张量分解方法及其在标签推荐中的应用

目前的标签推荐系统使用张量来存储"用户-资源-标签"三维数据,以挖掘三者之间潜在的语义关联。为更好地解决三维数据的稀疏性问题,避免张量填充造成的数据失真,提出基于标签惩罚机制的张量构建方法PM US和基于随机梯度下降的张量分解方法 HOSGD。利用标签惩罚机制和用户评分构建张量,使用随机梯度下降法对张量的展开矩阵进行分解。在此基础上,结合PMUS和HOSGD提出PMUS-HOSGD算法对数据进行处理,根据结果为用户进行个性化标签推荐。在数据集MovieLens上的实验结果表明,与CubeALS、HOSVD和CF算法相比,该算法能够有效提高标签推荐的准确率。

基于SGD的决策级融合维度情感识别方法

在双模态维度情感识别中,存在由于信息不全面而导致预测性能不高的缺陷,且使用决策级融合方法进行融合大多依赖支持向量回归算法,但该算法无法有效处理大样本情况。为了解决以上问题,在语音和文本模态的基础上增加动作捕捉(motion capture,Mocap)数据,并针对该多模态数据提出一种基于随机梯度下降(stochastic gradient descent,SGD)的决策级融合维度情感识别方法。结合多任务学习机制,利用不同的深度学习模型分别对语音、文本和Mocap特征进行训练,并基于决策级融合方法实现多模态维度情感识别。在IEMOCAP数据集上的实验结果表明,Mocap数据更有助于提高效价维的值,结合更多情感数据有助于提升维度情感识别的预测性能,基于SGD进行决策级融合得到的一致性相关系数均值高于其他回归算法。

分布式深度学习框架下基于性能感知的DBS-SGD算法

通过增加模型的深度以及训练数据的样本数量,深度神经网络模型能够在多个机器学习任务中获得更好的性能,然而这些必要的操作会使得深度神经网络模型训练的开销相应增大.因此为了更好地应对大量的训练开销,在分布式计算环境中对深度神经网络模型的训练过程进行加速成为了研发人员最常用的手段.随机梯度下降(stochastic gradient descent,SGD)算法是当前深度神经网络模型中最常见的训练算法之一,然而SGD在进行并行化的时候容易产生梯度过时问题,从而影响算法的整体收敛性.现有解决方案大部分针对的是各节点性能差别较小的高性能计算(high performance computing,HPC)环境,很少有研究考虑过各节点性能差别较大的集群环境.针对上述问题进行研究并提出了一种基于性能感知技术的动态batch size随机梯度下降算法(dynamic batch size SGD,DBS-SGD).该算法通过分析各节点的计算能力,对各节点的minibatch进行动态分配,从而保证了节点间每次迭代更新的时间基本一致,进而降低了节点的平均梯度过时值.提出的算法能够有效优化异步更新策略中存在的梯度过时问题.选用常用的图像分类基准Mnist和cifar10作为训练数据集,将该算法与异步随机梯度下降(asynchronous SGD,ASGD)算法、n-soft算法进行了对比.实验结果表明:在不损失加速比的情况下,Mnist数据集的loss函数值降低了60%,cifar数据集的准确率提升了约10%,loss函数值降低了10%,其性能高于ASGD算法和n-soft算法,接近同步策略下的收敛曲线.

基于SGDM优化IWOA-CNN的配电网工程造价控制研究

为了控制配电网工程项目的成本,需准确预测配电网工程造价,本文提出一种基于带动量因子的随机梯度下降(stochastic gradient descent with momentum factor, SGDM)优化的改进鲸鱼算法-卷积神经网络工程造价预测模型。首先,考虑回路数、杆塔数、导线、地形、地质、风速、覆冰、导线截面、混凝土杆、塔材、绝缘子(直线)、绝缘子(耐张)、基坑开方、基础钢材、底盘和水泥对配电网工程造价的影响,建立了非线性函数关系;采用SGDM优化器改进的卷积神经网络对函数进行逼近,并用贝叶斯方法优化卷积神经网络的超参数;利用改进的鲸鱼算法(improved whale optimization algorithm, IWOA)优化卷积神经网络,找出卷积神经网络的最优学习率。数值算例表明,新模型预测效果较好,并提出相应的控制策略。

基于Bagging-Down SGD算法的分布式深度网络

通过对大量数据进行训练并采用分布式深度学习算法可以学习到比较好的数据结构,而传统的分布式深度学习算法在处理大数据集时存在训练时间比较慢或者训练精度比较低的问题。提出Bootstrap向下聚合随机梯度下降(Bootstrap aggregating-down stochastic gradient descent,Bagging-Down SGD)算法重点来提高分布式深度网络的学习速率。Bagging-Down SGD算法通过在众多单机模型上加入速度控制器,对单机计算的参数值做统计处理,减少了参数更新的频率,并且可以使单机模型训练和参数更新在一定程度上分开,在保证训练精度的同时,提高了整个分布式模型的训练速度。该算法具有普适性,可以对多种类别的数据进行学习。

基于SGD算法优化的BP神经网络围岩参数反演模型研究

为充分利用现场监测数据所反馈的围岩变形信息,对岩体力学参数进行反演,以贵州省剑河至黎平高速公路TJ-1标段牛练塘隧道为工程背景,选择围岩弹性模量、黏聚力、泊松比及内摩擦角为影响因素,通过设计正交试验及有限元模拟,获取25组围岩物理力学参数组合及其对应的拱顶沉降值和拱腰收敛模拟值。基于随机梯度下降算法(stochastic gradient descent algorithm,简称SGD算法)对传统BP神经网络模型进行改进,建立以拱顶沉降值和拱腰收敛值为输入参数,以围岩弹性模量、黏聚力、泊松比及内摩擦角为输出值的基于SGD算法优化的BP神经网络模型,实现围岩参数的反演分析。将反演所得的围岩参数代入有限元模型,验证优化BP神经网络模型的可行性和准确性。最后,分析围岩变形及初期支护受力特性并给出施工建议。结果表明:1)基于SGD算法优化的BP神经网络模型计算得出的拱顶沉降值、拱腰收敛值、拱肩收敛值与现场实测值的相对误差率在2.50%~24.01%,均低于传统BP神经网络模型计算得出的误差率(11.51%~93.71%),验证优化BP神经网络模型的可行性和优越性;2)上、下台阶拱脚处的喷层和锚杆有应力集中现象,有破坏风险,建议施工中加强拱脚支护,防止发生工程事故。

Convergence of Stochastic Gradient Descent in Deep Neural Network

Stochastic gradient descent(SGD) is one of the most common optimization algorithms used in pattern recognition and machine learning.This algorithm and its variants are the preferred algorithm while optimizing parameters of deep neural network for their advantages of low storage space requirement and fast computation speed.Previous studies on convergence of these algorithms were based on some traditional assumptions in optimization problems.However,the deep neural network has its unique properties.Some assumptions are inappropriate in the actual optimization process of this kind of model.In this paper,we modify the assumptions to make them more consistent with the actual optimization process of deep neural network.Based on new assumptions,we studied the convergence and convergence rate of SGD and its two common variant algorithms.In addition,we carried out numerical experiments with LeNet-5,a common network framework,on the data set MNIST to verify the rationality of our assumptions.

A Primal-Dual SGD Algorithm for Distributed Nonconvex Optimization

The distributed nonconvex optimization problem of minimizing a global cost function formed by a sum of n local cost functions by using local information exchange is considered.This problem is an important component of many machine learning techniques with data parallelism,such as deep learning and federated learning.We propose a distributed primal-dual stochastic gradient descent(SGD)algorithm,suitable for arbitrarily connected communication networks and any smooth(possibly nonconvex)cost functions.We show that the proposed algorithm achieves the linear speedup convergence rate O(1/(√nT))for general nonconvex cost functions and the linear speedup convergence rate O(1/(nT)) when the global cost function satisfies the Polyak-Lojasiewicz(P-L)condition,where T is the total number of iterations.We also show that the output of the proposed algorithm with constant parameters linearly converges to a neighborhood of a global optimum.We demonstrate through numerical experiments the efficiency of our algorithm in comparison with the baseline centralized SGD and recently proposed distributed SGD algorithms.

基于Local SGD的部分同步通信策略

Local SGD训练方法用于分布式机器学习以缓解通信瓶颈,但其本地多轮迭代特性使异构集群节点计算时间差距增大,带来较大同步时延与参数陈旧问题。针对上述问题,基于Local SGD方法提出了一种动态部分同步通信策略(LPSP),该方法利用两层决策充分发挥Local SGD本地迭代优势。在节点每轮迭代计算结束后,基于本地训练情况判断通信可能性,并在全局划分同步集合以最小化同步等待时延,减少Local SGD通信开销并有效控制straggler负面影响。实验表明LPSP可以在不损失训练精确度的情况下实现最高0.75~1.26倍的加速,此外,最高还有5.14%的精确度提升,可以有效加速训练收敛。

A Stochastic Gradient Descent Method for Computational Design of Random Rough Surfaces in Solar Cells

In this work,we develop a stochastic gradient descent method for the computational optimal design of random rough surfaces in thin-film solar cells.We formulate the design problems as random PDE-constrained optimization problems and seek the optimal statistical parameters for the random surfaces.The optimizations at fixed frequency as well as at multiple frequencies and multiple incident angles are investigated.To evaluate the gradient of the objective function,we derive the shape derivatives for the interfaces and apply the adjoint state method to perform the computation.The stochastic gradient descent method evaluates the gradient of the objective function only at a few samples for each iteration,which reduces the computational cost significantly.Various numerical experiments are conducted to illustrate the efficiency of the method and significant increases of the absorptance for the optimal random structures.We also examine the convergence of the stochastic gradient descent algorithm theoretically and prove that the numerical method is convergent under certain assumptions for the random interfaces.

分布式训练系统及其优化算法综述

人工智能利用各种优化技术从海量训练样本中学习关键特征或知识以提高解的质量,这对训练方法提出了更高要求.然而,传统单机训练无法满足存储与计算性能等方面的需求.因此,利用多个计算节点协同的分布式训练系统成为热点研究方向之一.本文首先阐述了单机训练面临的主要挑战.其次,分析了分布式训练系统亟需解决的三个关键问题.基于上述问题归纳了分布式训练系统的通用框架与四个核心组件.围绕各个组件涉及的技术,梳理了代表性研究成果.在此基础之上,总结了基于并行随机梯度下降算法的中心化与去中心化架构研究分支,并对各研究分支优化算法与应用进行综述.最后,提出了未来可能的研究方向.

基于DSGD的分布式电磁目标识别

分布式电磁目标识别利用分布式最优化和分布式计算等技术实现传统集中式电磁目标识别。分布式最优化方法结合分布式计算架构实现对最优化问题的分布式求解,以分布式实现从问题信息、数据到最优目标识别模型的映射。利用去中心化随机梯度下降方法这一经典分布式最优化方法,建立面向电磁目标识别的分布式计算架构和分布式电磁目标识别方法。实际电磁信号数据验证了所提算法的有效性。在分布式电磁目标识别算法与集中式识别算法性能均保持在90%以上时,单节点训练时间下降50%以上,显著提升了训练效率。

基于FPGA并行实现SVM训练的可重构计算系统

针对支持向量机在处理大规模数据集时所面临的计算复杂度高和训练时间长的问题,设计了一种基于FPGA并行实现支持向量机训练的可重构计算系统,并分析了不同量化方式下的硬件资源消耗与加速性能。通过采用随机梯度下降法训练支持向量机,使得需要求解的维度与样本的维度相关联,相较于传统的基于二次规划的求解方法可以显著降低计算复杂性。同时,利用基于FPGA的可重构硬件平台设计了专用并行计算结构以加速支持向量机的训练过程。对设计的完整系统进行了软硬件联合仿真,在4个公共数据集上的仿真结果表明,整体模型预测准确率达到90%以上;在训练阶段,相较于采用相同算法的软件实现,所提出的浮点数表示下硬件实现的单个样本处理时间至少减少了2个数量级;定点数表示下硬件实现的单个样本处理时间最大减小了3个数量级;与基于二次规划问题求解的硬件实现相比,单个样本处理速度最快提升了394倍。

面向线束预装配的多约束线束布局方法研究

线束由一组线束段按树状结构连接而成,是飞机汽车等产品中连接各电气设备的接线部件。为提高安装效率,复杂线束需要在线束工装板图上进行预安装,即根据角度、距离、交叉、边界等工艺约束在工装板上摆放线束,是一个多约束下的线束布局问题。通过借鉴图布局算法,将线束布局转换为优化问题,并采用随机梯度下降法SGD每次随机挑选一对线束段进行优化,逐步迭代收敛。因为线束段是长度不变的刚体,所以移动一根线束段时会带动相连的其他线束段,进而导致SGD迭代过程出现震荡,难以收敛。通过提出双向传递的线束段移动算法,在保证线束段能移动到目标位置的同时尽量少相连线束段的变动。使用合成线束案例和某真实飞机线束案例进行有效性验证,结果表明各工艺约束能均能得到满足,符合线束预装配的生产要求。