Almost Sure Convergence of Proximal Stochastic Accelerated Gradient Methods

Proximal gradient descent and its accelerated version are resultful methods for solving the sum of smooth and non-smooth problems. When the smooth function can be represented as a sum of multiple functions, the stochastic proximal gradient method performs well. However, research on its accelerated version remains unclear. This paper proposes a proximal stochastic accelerated gradient (PSAG) method to address problems involving a combination of smooth and non-smooth components, where the smooth part corresponds to the average of multiple block sums. Simultaneously, most of convergence analyses hold in expectation. To this end, under some mind conditions, we present an almost sure convergence of unbiased gradient estimation in the non-smooth setting. Moreover, we establish that the minimum of the squared gradient mapping norm arbitrarily converges to zero with probability one.

A New Conjugate Gradient Projection Method for Solving Stochastic Generalized Linear Complementarity Problems

In this paper, a class of the stochastic generalized linear complementarity problems with finitely many elements is proposed for the first time. Based on the Fischer-Burmeister function, a new conjugate gradient projection method is given for solving the stochastic generalized linear complementarity problems. The global convergence of the conjugate gradient projection method is proved and the related numerical results are also reported.

Stochastic Finite Element Method for Mechanical Vibration Based on Conjugate Gradient(CG)

When material properties, geometry parameters and applied loads are assumed to be stochastic, the vibration equation of a system is transformed to static problem by using Newmark method. In order to improve the computational efficiency and to save storage, the Conjugate Gradient （CG） method is presented. The CG is an effective method for solving a large system of linear equations and belongs to the method of iteration with rapid convergence and high precision. An example is given and calculated results are compared to validate the proposed methods.

Feasibility of stochastic gradient boosting approach for predicting rockburst damage in burst-prone mines

The database of 254 rockburst events was examined for rockburst damage classification using stochastic gradient boosting （SGB） methods. Five potentially relevant indicators including the stress condition factor, the ground support system capacity, the excavation span, the geological structure and the peak particle velocity of rockburst sites were analyzed. The performance of the model was evaluated using a 10 folds cross-validation （CV） procedure with 80%of original data during modeling, and an external testing set （20%） was employed to validate the prediction performance of the SGB model. Two accuracy measures for multi-class problems were employed： classification accuracy rate and Cohen’s Kappa. The accuracy analysis together with Kappa for the rockburst damage dataset reveals that the SGB model for the prediction of rockburst damage is acceptable.

A Framework of Convergence Analysis of Mini-batch Stochastic Projected Gradient Methods

In this paper,we establish a unified framework to study the almost sure global convergence and the expected convergencerates of a class ofmini-batch stochastic(projected)gradient(SG)methods,including two popular types of SG:stepsize diminished SG and batch size increased SG.We also show that the standard variance uniformly bounded assumption,which is frequently used in the literature to investigate the convergence of SG,is actually not required when the gradient of the objective function is Lipschitz continuous.Finally,we show that our framework can also be used for analyzing the convergence of a mini-batch stochastic extragradient method for stochastic variational inequality.

A Stochastic Gradient Descent Method for Computational Design of Random Rough Surfaces in Solar Cells

In this work,we develop a stochastic gradient descent method for the computational optimal design of random rough surfaces in thin-film solar cells.We formulate the design problems as random PDE-constrained optimization problems and seek the optimal statistical parameters for the random surfaces.The optimizations at fixed frequency as well as at multiple frequencies and multiple incident angles are investigated.To evaluate the gradient of the objective function,we derive the shape derivatives for the interfaces and apply the adjoint state method to perform the computation.The stochastic gradient descent method evaluates the gradient of the objective function only at a few samples for each iteration,which reduces the computational cost significantly.Various numerical experiments are conducted to illustrate the efficiency of the method and significant increases of the absorptance for the optimal random structures.We also examine the convergence of the stochastic gradient descent algorithm theoretically and prove that the numerical method is convergent under certain assumptions for the random interfaces.

基于算力-能量全分布式在线共享的5G网络负荷管理策略

5G与边缘计算等信息基础设施海量部署造成运营商用电成本上升,需推动边缘网络与电网的能量互动以节能降本。现有研究重点关注边缘网络参与日前经济调度,未考虑可再生能源和网络流量双重随机性造成的网络能量供需不平衡问题。针对强随机环境下的网络负荷管理问题,提出面向虚拟化边缘网络的能量实时管理策略。首先,以网络用能成本最小化为目标,构建联合网络资源管理、储能充放电与能量共享模型。其次,针对未来网络信息未知无法直接求解的问题,提出基于随机对偶次梯度法的在线管理策略。然后,针对资源共享涉及运营商隐私问题,提出全分布式的计算资源与能量协同共享算法。最后,仿真验证表明,所提在线算法在无需先验知识的前提下有效减少了5G边缘网络的用能成本。

求解一类非光滑凸优化问题的相对加速SGD算法

一阶优化算法由于其计算简单、代价小,被广泛应用于机器学习、大数据科学、计算机视觉等领域,然而,现有的一阶算法大多要求目标函数具有Lipschitz连续梯度,而实际中的很多应用问题不满足该要求。在经典的梯度下降算法基础上,引入随机和加速,提出一种相对加速随机梯度下降算法。该算法不要求目标函数具有Lipschitz连续梯度,而是通过将欧氏距离推广为Bregman距离,从而将Lipschitz连续梯度条件减弱为相对光滑性条件。相对加速随机梯度下降算法的收敛性与一致三角尺度指数有关,为避免调节最优一致三角尺度指数参数的工作量,给出一种自适应相对加速随机梯度下降算法。该算法可自适应地选取一致三角尺度指数参数。对算法收敛性的理论分析表明,算法迭代序列的目标函数值收敛于最优目标函数值。针对Possion反问题和目标函数的Hessian阵算子范数随变量范数多项式增长的极小化问题的数值实验表明,自适应相对加速随机梯度下降算法和相对加速随机梯度下降算法的收敛性能优于相对随机梯度下降算法。

A Mini-Batch Proximal Stochastic Recursive Gradient Algorithm with Diagonal Barzilai–Borwein Stepsize

Many machine learning problems can be formulated as minimizing the sum of a function and a non-smooth regularization term.Proximal stochastic gradient methods are popular for solving such composite optimization problems.We propose a minibatch proximal stochastic recursive gradient algorithm SRG-DBB,which incorporates the diagonal Barzilai–Borwein(DBB)stepsize strategy to capture the local geometry of the problem.The linear convergence and complexity of SRG-DBB are analyzed for strongly convex functions.We further establish the linear convergence of SRGDBB under the non-strong convexity condition.Moreover,it is proved that SRG-DBB converges sublinearly in the convex case.Numerical experiments on standard data sets indicate that the performance of SRG-DBB is better than or comparable to the proximal stochastic recursive gradient algorithm with best-tuned scalar stepsizes or BB stepsizes.Furthermore,SRG-DBB is superior to some advanced mini-batch proximal stochastic gradient methods.

Least-Squares Seismic Inversion with Stochastic Conjugate Gradient Method

With the development of computational power, there has been an increased focus on data-fitting related seismic inversion techniques for high fidelity seismic velocity model and image, such as full-waveform inversion and least squares migration. However, though more advanced than conventional methods, these data fitting methods can be very expensive in terms of computational cost. Recently, various techniques to optimize these data-fitting seismic inversion problems have been implemented to cater for the industrial need for much improved efficiency. In this study, we propose a general stochastic conjugate gradient method for these data-fitting related inverse problems. We first prescribe the basic theory of our method and then give synthetic examples. Our numerical experiments illustrate the potential of this method for large-size seismic inversion application.

Performance Enhancement of Adaptive Neural Networks Based on Learning Rate

Deep learning is the process of determining parameters that reduce the cost function derived from the dataset.The optimization in neural networks at the time is known as the optimal parameters.To solve optimization,it initialize the parameters during the optimization process.There should be no variation in the cost function parameters at the global minimum.The momentum technique is a parameters optimization approach;however,it has difficulties stopping the parameter when the cost function value fulfills the global minimum(non-stop problem).Moreover,existing approaches use techniques;the learning rate is reduced during the iteration period.These techniques are monotonically reducing at a steady rate over time;our goal is to make the learning rate parameters.We present a method for determining the best parameters that adjust the learning rate in response to the cost function value.As a result,after the cost function has been optimized,the process of the rate Schedule is complete.This approach is shown to ensure convergence to the optimal parameters.This indicates that our strategy minimizes the cost function(or effective learning).The momentum approach is used in the proposed method.To solve the Momentum approach non-stop problem,we use the cost function of the parameter in our proposed method.As a result,this learning technique reduces the quantity of the parameter due to the impact of the cost function parameter.To verify that the learning works to test the strategy,we employed proof of convergence and empirical tests using current methods and the results are obtained using Python.

ON EFFECTIVE STOCHASTIC GALERKIN FINITE ELEMENT METHOD FOR STOCHASTIC OPTIMAL CONTROL GOVERNED BY INTEGRAL-DIFFERENTIAL EQUATIONS WITH RANDOM COEFFICIENTS

In this paper, we apply stochastic Galerkin finite element methods to the optimal control problem governed by an elliptic integral-differential PDEs with random field. The control problem has the control constraints of obstacle type. A new gradient algorithm based on the pre-conditioner conjugate gradient algorithm （PCG） is developed for this optimal control problem. This algorithm can transform a part of the state equation matrix and co-state equation matrix into block diagonal matrix and then solve the optimal control systems iteratively. The proof of convergence for this algorithm is also discussed. Finally numerical examples of a medial size are presented to illustrate our theoretical results.

A FAST STOCHASTIC GALERKIN METHOD FOR A CONSTRAINED OPTIMAL CONTROL PROBLEM GOVERNED BY A RANDOM FRACTIONAL DIFFUSION EQUATION

We develop a fast stochastic Galerkin method for an optimal control problem governed by a random space-fractional diffusion equation with deterministic constrained control. Optimal control problems governed by a fractional diffusion equation tends to provide a better description for transport or conduction processes in heterogeneous media. Howev- er, the fractional control problem introduces significant computation complexity due to the nonlocal nature of fractional differential operators, and this is further worsen by the large number of random space dimensions to discretize the probability space. We ap- proximate the optimality system by a gradient algorithm combined with the stochastic Galerkin method through the discretization with respect to both the spatial space and the probability space. The resulting linear system can be decoupled for the random and spatial variable, and thus solved separately. A fast preconditioned Bi-Conjugate Gradient Stabilized method is developed to efficiently solve the decoupled systems derived from the fractional diffusion operators in the spatial space. Numerical experiments show the utility of the method.

带有随机改进Barzilai-Borwein步长的小批量稀疏随机方差缩减梯度法

为了更好地应对当今时代的大规模高维稀疏数据集,融合BB方法、小批量算法与随机方差缩减梯度法(SVRG)优势,提出一种带有随机改进Barzilai-Borwein步长的小批量稀疏随机方差缩减梯度法(MSSVRG-R2BB)。首先,在SVRG外循环中全梯度计算的基础上加入L_1范数次梯度设计出一种稀疏近似梯度用于内循环,得到一种稀疏的SVRG算法(SSVRG)。在此基础上,在小批量的稀疏随机方差缩减梯度法中使用随机选取的改进BB方法自动计算、更新步长,解决了小批量算法的步长选取问题,拓展得到MSSVRG-R2BB算法。数值实验表明,在求解大规模高维稀疏数据的线性支持向量机(SVM)问题时,MSSVRG-R2BB算法不仅可以减小运算成本、更快达到收敛上界,同时能达到与其他先进的小批量算法相同的优化水平,并且对于不同的初始参数选取表现稳定且良好。

基于块坐标下降法的神经网络学习算法

针对当前一阶优化算法收敛速度慢,对学习率依赖性强和二阶优化方法复杂度高等问题,利用神经网络固有的可分性,提出递归最小二乘与梯度下降的混合算法(Hybrid Recursive Least-Squares with Stochastic Gradient Descent, HRLSGD),将原本复杂的网络模型分解为更易解决的低维优化问题。实验结果表明,HRLSGD的收敛速度优于主流的一阶优化算法,对于学习率的鲁棒性更高。

考虑参数变异性的土石坝渗透破坏概率分析

为合理考虑土体参数变异性对土石坝渗透破坏概率的影响,本文编写了基于GEOSTUDIO与MATLAB之间的渗流有限元计算及可靠度分析的接口实现程序,并以一实际土石坝工程为例验证了提出方法的有效性,探讨了库水位骤降条件下坝体材料渗透系数以及土水特征曲线模型参数的变异性对土石坝渗透破坏概率的影响。结果表明:库水位上升对土石坝渗透稳定很不利,库水位从正常蓄水位工况上升至校核洪水位工况,土石坝渗透破坏概率增加近两倍。受坝体材料基质吸力的作用,库水位骤降会导致坝体发生逆向渗流,土石坝渗透破坏概率到达某一时刻后会急剧增加,然后直至坝体内孔隙水压力得到充分消散才下降并趋于某一稳定值。

随机共轭梯度反演法

本文提出一种新的非线性反演方法——随机共轭梯度法。该方法采用非启发式反演方法 ,快速收敛到某一极值 ;再用启发式反演方法跳出局部极值 ;然后使用非启发式反演方法收敛到另一局部极值 ,反复进行此过程 ;并在解空间范围内搜索 ,保留所有的局部极值 ,最终确定最优解。它继承了随机爬山能够全局寻优、共轭梯度法计算速度快和精度高的优点 ,能快速搜索到全局最优解。试验证明 ,这种方法是一种高效的反演算法 ,特别适用于求解非线性、多极值的最优化问题和地球物理反问题。

基于复合卷积神经网络的图像去噪算法

基于深度学习理论,将图像去噪过程看成神经网络的拟合过程,构造简洁高效的复合卷积神经网络,提出基于复合卷积神经网络的图像去噪算法.算法第1阶段由2个2层的卷积网络构成,分别训练阶段2中的3层卷积网络中的部分初始卷积核,缩短阶段2中网络的训练时间和增强算法的鲁棒性.最后运用阶段2中的卷积网络对新的噪声图像进行有效去噪.实验表明文中算法在峰值信噪比、结构相识度及均方根误差指数上与当前较好的图像去噪算法相当,尤其当噪声加强时效果更佳且训练时间较短.

一种带有梯度加速的粒子群算法

通过引入梯度信息来影响粒子速度的更新,构造了一种带有梯度加速的粒子群算法.为减小陷入局优的可能性,当群体最优信息陷入停滞时,对群体进行部分初始化来保持群体的活性,并讨论了改进算法的适用范围.仿真结果表明,对于单峰函数和多峰函数,改进算法都能够取得较好的优化效果.

随机梯度下降法的一些性质(英文)

本文研究了一般核空间下的随机梯度下降法.通过迭代方法,给出了该算法的一些重要性质,这些性质对于研究收敛速度起到至关重要的作用.