Almost Sure Convergence of Proximal Stochastic Accelerated Gradient Methods

Proximal gradient descent and its accelerated version are resultful methods for solving the sum of smooth and non-smooth problems. When the smooth function can be represented as a sum of multiple functions, the stochastic proximal gradient method performs well. However, research on its accelerated version remains unclear. This paper proposes a proximal stochastic accelerated gradient (PSAG) method to address problems involving a combination of smooth and non-smooth components, where the smooth part corresponds to the average of multiple block sums. Simultaneously, most of convergence analyses hold in expectation. To this end, under some mind conditions, we present an almost sure convergence of unbiased gradient estimation in the non-smooth setting. Moreover, we establish that the minimum of the squared gradient mapping norm arbitrarily converges to zero with probability one.

GLOBAL CONVERGENCE RESULTS OF A THREE TERM MEMORY GRADIENT METHOD WITH A NON-MONOTONE LINE SEARCH TECHNIQUE

In this paper, a new class of three term memory gradient method with non-monotone line search technique for unconstrained optimization is presented. Global convergence properties of the new methods are discussed. Combining the quasi-Newton method with the new method, the former is modified to have global convergence property. Numerical results show that the new algorithm is efficient.

Preparation of berberine hydrochloride long-circulating liposomes by ionophore A23187-mediated ZnSO_(4)gradient method

The aim of the study was to prepare berberine hydrochloride long-circulating liposomes and optimize the formulation and process parameters,and investigate the influence of different factors on the encapsulation efficiency.Berberine hydrochloride liposomes were prepared in response to a transmembrane ion gradient that was established by ionophore A23187.Free and liposomal drug were separated by cation exchange resin,and then the amount of intraliposomal berberine hydrochloride was determined by UV spectrophotometry.The optimized encapsulation efficiency of berberine hydrochloride liposomes was 94.3%
2.1%when the drug-to-lipid ratio was 1:20,and the mean diameter was 146.9 nm
3.2 nm.As a result,the ionophore A23187-mediated ZnSO_(4)gradient method was suitable for the preparation of berberine hydrochloride liposomes that we could get the desired encapsulation efficiency and drug loading.

Convergence of Online Gradient Method with Penalty for BP Neural Networks

Online gradient method has been widely used as a learning algorithm for training feedforward neural networks. Penalty is often introduced into the training procedure to improve the generalization performance and to decrease the magnitude of network weights. In this paper, some weight boundedness and deterministic con- vergence theorems are proved for the online gradient method with penalty for BP neural network with a hidden layer, assuming that the training samples are supplied with the network in a fixed order within each epoch. The monotonicity of the error function with penalty is also guaranteed in the training iteration. Simulation results for a 3-bits parity problem are presented to support our theoretical results.

A modified three–term conjugate gradient method with sufficient descent property

A hybridization of the three–term conjugate gradient method proposed by Zhang et al. and the nonlinear conjugate gradient method proposed by Polak and Ribi`ere, and Polyak is suggested. Based on an eigenvalue analysis, it is shown that search directions of the proposed method satisfy the sufficient descent condition, independent of the line search and the objective function convexity. Global convergence of the method is established under an Armijo–type line search condition. Numerical experiments show practical efficiency of the proposed method.

IMPROVED GRADIENT METHOD FOR MONOTONE AND LIPSCHITZ CONTINUOUS MAPPINGS IN BANACH SPACES

Let C be a nonempty closed convex subset of a 2-uniformly convex and uniformly smooth Banach space E and {An}n∈N be a family of monotone and Lipschitz continuos mappings of C into E＊. In this article, we consider the improved gradient method by the hybrid method in mathematical programming [i0] for solving the variational inequality problem for {AN} and prove strong convergence theorems. And we get several results which improve the well-known results in a real 2-uniformly convex and uniformly smooth Banach space and a real Hilbert space.

High-efciency improved symmetric successive over-relaxation preconditioned conjugate gradient method for solving large-scale finite element linear equations

Fast solving large-scale linear equations in the finite element analysis is a classical subject in computational mechanics. It is a key technique in computer aided engineering （CAE） and computer aided manufacturing （CAM）. This paper presents a high-efficiency improved symmetric successive over-relaxation （ISSOR） preconditioned conjugate gradient （PCG） method, which maintains lelism consistent with the original form. Ideally, the by 50% as compared with the original algorithm. the convergence and inherent paralcomputation can It is suitable for be reduced nearly high-performance computing with its inherent basic high-efficiency operations. By comparing with the numerical results, it is shown that the proposed method has the best performance.

One Step Positive Approach for Sheet Metal Forming Simulation Based on Quasi-conjugate-gradient Method

In one step inverse finite element approach, an initial blank shape is normally predicted from the final deformed shape. The final deformed shape needs to be trimmed into a final part after stamping, the trimmed area, therefore, needs to be compensated manually before using one step inverse approach, which causes low efficiency and in consistency with the real situation. To solve this problem, one step positive approach is proposed to simulate the sheet metal stamping process. Firstly the spatial initial solution of one step positive method is preliminarily obtained by using the mapping relationship and area coordinates, then based on the deformation theory the iterative solving is carried out in three-dimensional coordinate system by using quasi-conjugate-gradient method. During iterative process the contact judgment method is introduced to ensure that the nodes on the spatial initial solution are not separated from die surface. The predicted results of sheet metal forming process that include the shape and thickness of the stamped part can be obtained after the iterative solving process. The validity of the proposed approach is verified by comparing the predicted results obtained through the proposed approach with those obtained through the module of one step inverse approach in Autoform and the real stamped part. In one step positive method, the stamped shape of regular sheet can be calculated fast and effectively. During the iterative solution, the quasi-conjugate-gradient method is proposed to take the place of solving system of equations, and it can improve the stability and precision of the algorithm.

Online Gradient Methods with a Punishing Term for Neural Networks

Online gradient methods are widely used for training the weight of neural networks and for other engineering computations. In certain cases, the resulting weight may become very large, causing difficulties in the implementation of the network by electronic circuits. In this paper we introduce a punishing term into the error function of the training procedure to prevent this situation. The corresponding convergence of the iterative training procedure and the boundedness of the weight sequence are proved. A supporting numerical example is also provided.

Subspace Minimization Conjugate Gradient Method Based on Cubic Regularization Model for Unconstrained Optimization

Many methods have been put forward to solve unconstrained optimization problems,among which conjugate gradient method(CG)is very important.With the increasing emergence of large⁃scale problems,the subspace technology has become particularly important and widely used in the field of optimization.In this study,a new CG method was put forward,which combined subspace technology and a cubic regularization model.Besides,a special scaled norm in a cubic regularization model was analyzed.Under certain conditions,some significant characteristics of the search direction were given and the convergence of the algorithm was built.Numerical comparisons show that for the 145 test functions under the CUTEr library,the proposed method is better than two classical CG methods and two new subspaces conjugate gradient methods.

An Efficient Projected Gradient Method for Convex Constrained Monotone Equations with Applications in Compressive Sensing

In this paper, a modified Polak-Ribière-Polyak conjugate gradient projection method is proposed for solving large scale nonlinear convex constrained monotone equations based on the projection method of Solodov and Svaiter. The obtained method has low-complexity property and converges globally. Furthermore, this method has also been extended to solve the sparse signal reconstruction in compressive sensing. Numerical experiments illustrate the efficiency of the given method and show that such non-monotone method is suitable for some large scale problems.

A New Class of Nonlinear Conjugate Gradient Methods with Global Convergence Properties

非线性共轭梯度法由于其迭代简单和储存量小,且搜索方向不需要满足正割条件,在求解大规模无约束优化问题时占据及其重要的地位.提出了一类新的共轭梯度法,其搜索方向是目标函数的下降方向.若假设目标函数连续可微且梯度满足Lipschitz条件,线性搜索满足Wolfe原则,讨论了所设计算法的全局收敛性.

GLOBAL CONVERGENCE OF THE GENERAL THREE TERM CONJUGATE GRADIENT METHODS WITH THE RELAXED STRONG WOLFE LINE SEARCH

The global convergence of the general three term conjugate gradient methods with the relaxed strong Wolfe line search is proved.

IMPROVED PRECONDITIONED CONJUGATE GRADIENT METHOD AND ITS APPLICATION IN F.E.A.FOR ENGINEERING

In this paper two theorems with theoretical and practical significance are given in respect to the preconditioned conjugate gradient method (PCCG). The theorems discuss respectively the qualitative property of the iterative solution and the construction principle of the iterative matrix. The authors put forward a new incompletely LU factorizing technique for non-M-matrix and the method of constructing the iterative matrix. This improved PCCG is used to calculate the ill-conditioned problems and large-scale three-dimensional finite element problems, and simultaneously contrasted with other methods. The abnormal phenomenon is analyzed when PCCG is used to solve the system of ill-conditioned equations, ft is shown that the method proposed in this paper is quite effective in solving the system of large-scale finite element equations and the system of ill-conditioned equations.

A Descent Gradient Method and Its Global Convergence

Y Liu and C Storey(1992)proposed the famous LS conjugate gradient method which has good numerical results.However,the LS method has very weak convergence under the Wolfe-type line search.In this paper,we give a new descent gradient method based on the LS method.It can guarantee the sufficient descent property at each iteration and the global convergence under the strong Wolfe line search.Finally,we also present extensive preliminary numerical experiments to show the efficiency of the proposed method by comparing with the famous PRP^+method.

A note on a family of proximal gradient methods for quasi-static incremental problems in elastoplastic analysis

Accelerated proximal gradient methods have recently been developed for solving quasi-static incremental problems of elastoplastic analysis with some different yield criteria.It has been demonstrated through numerical experiments that these methods can outperform conventional optimization-based approaches in computational plasticity.However,in literature these algorithms are described individually for specific yield criteria,and hence there exists no guide for application of the algorithms to other yield criteria.This short paper presents a general form of algorithm design,independent of specific forms of yield criteria,that unifies the existing proximal gradient methods.Clear interpretation is also given to each step of the presented general algorithm so that each update rule is linked to the underlying physical laws in terms of mechanical quantities.

A CLASSOF NONMONOTONE CONJUGATE GRADIENT METHODSFOR NONCONVEX FUNCTIONS

This paper discusses the global convergence of a class of nonmonotone conjugate gra- dient methods(NM methods) for nonconvex object functions.This class of methods includes the nonmonotone counterpart of modified Polak- Ribière method and modified Hestenes- Stiefel method as special cases

APPLICATION OF HIGH-RESOLUTION MAGNETIC METHOD AND GRADIENT METHOD TO LOCATE ABANDONED BRINE-WELLS IN HUTCHINSON, KANSAS, U.S.A.

After successfully locating one abandoned brine well by an electromagnetic method during testing in 2001 and five abandoned brine wells by a high resolution magnetic method during 2002, a high resolution magnetic method was again proposed to search for wells in 2003 when a second sensor was employed to acquire data for calculating the pseudo vertical gradient of magnetic fields. Total area surveyed in 2003 was 1,024,000 ft 2, which was divided into grids with an average size of 10,000 ft 2 and distributed across eight different sites. Magnetic anomalies and their vertical gradients from known brine wells were first recorded as signatures to identify anomalies caused by possible buried brine wells. Of fifty one verified anomalies, thirty one anomalies were due to wells buried at depths from 0 to 8.5 ft: twenty one 6 to 9 inch abandoned brine wells, seven 1.5 to 3 inch probable water wells, one 16 inch dewatering well for a construction site at a depth of 3 ft, and two 4 inch wells on the ground surface. Approximate monopole shaped anomalies were observed from all these wells after data corrections. However, the range of amplitudes of magnetic anomalies from 7,000 to 28,000 nT from these abandoned brine wells was measured. This range of anomalies is mainly due to the thickness and depth of buried wells. Anomaly amplitudes from 1.5 to 3 inch wells are 4,000 to 8,000 nT and linearly correlate with the buried depth. One 3 inch well that caused an anomaly of 13,000 nT could be the inner pipe of a brine well. Gradient anomalies are roughly in a range of 100 to 200 nT/inch for 1.5 to 3 inch wells and 200 to 300 nT/inch for brine wells.As indicated by the potential field theory, gradient data possess higher horizontal resolution than the magnetic field itself. Gradient data provide valuable assistance in determining horizontal locations of anomaly sources for excavation. In practice, however, improvement in the horizontal resolution is limited by survey line spacing. If only one sensor is used in a survey, there is rapid decrease in the horizontal resolution when sensor height increases from 14 to 44 inches. This indicates that it is critical to keep the sensor as close to the ground as possible when hunting buried wells that are close to each other. It also suggests that the downward continuation is useful to increase the horizontal resolution in well hunting.

Application of the Generalized Reduced Gradient Method in Mechanical Optimization Design

This paper presents the generalized reduced gradient method （GRG） and its realization forms. The application example of GRG in the optimization design of a single-stage cylindrical gear reducer is introduced. The algo- rithm of the GRG method is realized in Vissim software. Based on the mathematical model of the single-stage cylin- drical gear reducer, the simulation structure of the optimization design was achieved. The experiment results show that the GRG method has fewer iterations and higher precision. The GRG method is very suitable for solving mechanical optimization design.

CONVERGENCE ANALYSIS ON A CLASS OF CONJUGATE GRADIENT METHODS WITHOUTSUFFICIENT DECREASE CONDITION

Recently, Gilbert and Nocedal([3]) investigated global convergence of conjugate gradient methods related to Polak-Ribiere formular, they restricted beta(k) to non-negative value. [5] discussed the same problem as that in [3] and relaxed beta(k) to be negative with the objective function being convex. This paper allows beta(k) to be selected in a wider range than [5]. Especially, the global convergence of the corresponding algorithm without sufficient decrease condition is proved.