Further study on a class of augmented Lagrangians of Di Pillo and Grippo in nonlinear programming

In this paper, a class of augmented Lagrangiaus of Di Pillo and Grippo （DGALs） was considered, for solving equality-constrained problems via unconstrained minimization techniques. The relationship was further discussed between the uneonstrained minimizers of DGALs on the product space of problem variables and multipliers, and the solutions of the eonstrained problem and the corresponding values of the Lagrange multipliers. The resulting properties indicate more precisely that this class of DGALs is exact multiplier penalty functions. Therefore, a solution of the equslity-constralned problem and the corresponding values of the Lagrange multipliers can be found by performing a single unconstrained minimization of a DGAL on the product space of problem variables and multipliers.

EXACT AUGMENTED LAGRANGIAN FUNCTION FOR NONLINEAR PROGRAMMING PROBLEMS WITH INEQUALITY CONSTRAINTS

An exact augmented Lagrangian function for the nonlinear nonconvex programming problems with inequality constraints was discussed. Under suitable hypotheses, the relationship was established between the local unconstrained minimizers of the augmented Lagrangian function on the space of problem variables and the local minimizers of the original constrained problem. Furthermore, under some assumptions, the relationship was also established between the global solutions of the augmented Lagrangian function on some compact subset of the space of problem variables and the global solutions of the constrained problem. Therefore, f^om the theoretical point of view, a solution of the inequality constrained problem and the corresponding values of the Lagrange multipliers can be found by the well-known method of multipliers which resort to the unconstrained minimization of the augmented Lagrangian function presented.

An Optimization Model for the Strip-packing Problem and Its Augmented Lagrangian Method

This paper formulates a two-dimensional strip packing problem as a non- linear programming （NLP） problem and establishes the first-order optimality conditions for the NLP problem. A numerical algorithm for solving this NLP problem is given to find exact solutions to strip-packing problems involving up to 10 items. Approximate solutions can be found for big-sized problems by decomposing the set of items into small-sized blocks of which each block adopts the proposed numerical algorithm. Numerical results show that the approximate solutions to big-sized problems obtained by this method are superior to those by NFDH, FFDH and BFDH approaches.

An accelerated augmented Lagrangian method for linearly constrained convex programming with the rate of convergence O(1/k^2)

In this paper, we propose and analyze an accelerated augmented Lagrangian method（denoted by AALM） for solving the linearly constrained convex programming. We show that the convergence rate of AALM is O（1/k^2） while the convergence rate of the classical augmented Lagrangian method（ALM） is O1 k. Numerical experiments on the linearly constrained 1-2minimization problem are presented to demonstrate the effectiveness of AALM.

Approximate Augmented Lagrangian Functions and Nonlinear Semidefinite Programs

In this paper, an approximate augmented Lagrangian function for nonlinear semidefinite programs is introduced. Some basic properties of the approximate augmented Lagrange function such as monotonicity and convexity are discussed. Necessary and sufficient conditions for approximate strong duality results are derived. Conditions for an approximate exact penalty representation in the framework of augmented Lagrangian are given. Under certain conditions, it is shown that any limit point of a sequence of stationary points of approximate augmented Lagrangian problems is a KKT point of the original semidefinite program and that a sequence of optimal solutions to augmented Lagrangian problems converges to a solution of the original semidefinite program.

A Fast Augmented Lagrangian Method for Euler’s Elastica Models

In this paper,a fast algorithm for Euler’s elastica functional is proposed,in which the Euler’s elastica functional is reformulated as a constrained minimization problem.Combining the augmented Lagrangian method and operator splitting techniques,the resulting saddle-point problem is solved by a serial of subproblems.To tackle the nonlinear constraints arising in the model,a novel fixed-point-based approach is proposed so that all the subproblems either is a linear problem or has a closed-form solution.We show the good performance of our approach in terms of speed and reliability using numerous numerical examples on synthetic,real-world and medical images for image denoising,image inpainting and image zooming problems.

Multi-scale UDCT dictionary learning based highly undersampled MR image reconstruction using patch-based constraint splitting augmented Lagrangian shrinkage algorithm

Recently, dictionary learning(DL) based methods have been introduced to compressed sensing magnetic resonance imaging(CS-MRI), which outperforms pre-defined analytic sparse priors. However, single-scale trained dictionary directly from image patches is incapable of representing image features from multi-scale, multi-directional perspective, which influences the reconstruction performance. In this paper, incorporating the superior multi-scale properties of uniform discrete curvelet transform(UDCT) with the data matching adaptability of trained dictionaries, we propose a flexible sparsity framework to allow sparser representation and prominent hierarchical essential features capture for magnetic resonance(MR) images. Multi-scale decomposition is implemented by using UDCT due to its prominent properties of lower redundancy ratio, hierarchical data structure, and ease of implementation. Each sub-dictionary of different sub-bands is trained independently to form the multi-scale dictionaries. Corresponding to this brand-new sparsity model, we modify the constraint splitting augmented Lagrangian shrinkage algorithm(C-SALSA) as patch-based C-SALSA(PB C-SALSA) to solve the constraint optimization problem of regularized image reconstruction. Experimental results demonstrate that the trained sub-dictionaries at different scales, enforcing sparsity at multiple scales, can then be efficiently used for MRI reconstruction to obtain satisfactory results with further reduced undersampling rate. Multi-scale UDCT dictionaries potentially outperform both single-scale trained dictionaries and multi-scale analytic transforms. Our proposed sparsity model achieves sparser representation for reconstructed data, which results in fast convergence of reconstruction exploiting PB C-SALSA. Simulation results demonstrate that the proposed method outperforms conventional CS-MRI methods in maintaining intrinsic properties, eliminating aliasing, reducing unexpected artifacts, and removing noise. It can achieve comparable performance of reconstruction with the state-of-the-art methods even under substantially high undersampling factors.

Fast Linearized Augmented Lagrangian Method for Euler’s Elastica Model

Recently,many variational models involving high order derivatives have been widely used in image processing,because they can reduce staircase effects during noise elimination.However,it is very challenging to construct efficient algo-rithms to obtain the minimizers of original high order functionals.In this paper,we propose a new linearized augmented Lagrangian method for Euler’s elastica image denoising model.We detail the procedures of finding the saddle-points of the aug-mented Lagrangian functional.Instead of solving associated linear systems by FFTor linear iterative methods(e.g.,the Gauss-Seidel method),we adopt a linearized strat-egy to get an iteration sequence so as to reduce computational cost.In addition,we give some simple complexity analysis for the proposed method.Experimental results with comparison to the previous method are supplied to demonstrate the efficiency of the proposed method,and indicate that such a linearized augmented Lagrangian method is more suitable to deal with large-sized images.

AN AUGMENTED LAGRANGIAN TRUST REGION METHOD WITH A BI-OBJECT STRATEGY

An augmented Lagrangian trust region method with a bi=object strategy is proposed for solving nonlinear equality constrained optimization, which falls in between penalty-type methods and penalty-free ones. At each iteration, a trial step is computed by minimizing a quadratic approximation model to the augmented Lagrangian function within a trust region. The model is a standard trust region subproblem for unconstrained optimization and hence can efficiently be solved by many existing methods. To choose the penalty parameter, an auxiliary trust region subproblem is introduced related to the constraint violation. It turns out that the penalty parameter need not be monotonically increasing and will not tend to infinity. A bi-object strategy, which is related to the objective function and the measure of constraint violation, is utilized to decide whether the trial step will be accepted or not. Global convergence of the method is established under mild assumptions. Numerical experiments are made, which illustrate the efficiency of the algorithm on various difficult situations.

Augmented Lagrangian Methods for p-Harmonic Flows with the Generalized Penalization Terms and Application to Image Processing

In this paper,we propose a generalized penalization technique and a convex constraint minimization approach for the p-harmonic flow problem following the ideas in[Kang&March,IEEE T.Image Process.,16(2007),2251–2261].We use fast algorithms to solve the subproblems,such as the dual projection methods,primal-dual methods and augmented Lagrangian methods.With a special penalization term,some special algorithms are presented.Numerical experiments are given to demonstrate the performance of the proposed methods.We successfully show that our algorithms are effective and efficient due to two reasons:the solver for subproblem is fast in essence and there is no need to solve the subproblem accurately(even 2 inner iterations of the subproblem are enough).It is also observed that better PSNR values are produced using the new algorithms.

Augmented Lagrangian Methods for Convex Matrix Optimization Problems

In this paper,we provide some gentle introductions to the recent advance in augmented Lagrangian methods for solving large-scale convex matrix optimization problems(cMOP).Specifically,we reviewed two types of sufficient conditions for ensuring the quadratic growth conditions of a class of constrained convex matrix optimization problems regularized by nonsmooth spectral functions.Under a mild quadratic growth condition on the dual of cMOP,we further discussed the R-superlinear convergence of the Karush-Kuhn-Tucker(KKT)residuals of the sequence generated by the augmented Lagrangian methods(ALM)for solving convex matrix optimization problems.Implementation details of the ALM for solving core convex matrix optimization problems are also provided.

Newton-conjugate gradient (CG) augmented Lagrangian method for path constrained dynamic process optimization

In this paper, a Newton-conjugate gradient （CG） augmented Lagrangian method is proposed for solving the path constrained dynamic process optimization problems. The path constraints are simplified as a single final time constraint by using a novel constraint aggregation function. Then, a control vector parameterization （CVP） approach is applied to convert the constraints simplified dynamic optimization problem into a nonlinear programming （NLP） problem with inequality constraints. By constructing an augmented Lagrangian function, the inequality constraints are introduced into the augmented objective function, and a box constrained NLP problem is generated. Then, a linear search Newton-CG approach, also known as truncated Newton （TN） approach, is applied to solve the problem. By constructing the Hamiltonian functions of objective and constraint functions, two adjoint systems are generated to calculate the gradients which are needed in the process of NLP solution. Simulation examlales demonstrate the effectiveness of the algorithm.

An Augmented Lagrangian Deep Learning Method for Variational Problems with Essential Boundary Conditions

This paper is concerned with a novel deep learning method for variational problems with essential boundary conditions.To this end,wefirst reformulate the original problem into a minimax problem corresponding to a feasible augmented La-grangian,which can be solved by the augmented Lagrangian method in an infinite dimensional setting.Based on this,by expressing the primal and dual variables with two individual deep neural network functions,we present an augmented Lagrangian deep learning method for which the parameters are trained by the stochastic optimiza-tion method together with a projection technique.Compared to the traditional penalty method,the new method admits two main advantages:i)the choice of the penalty parameter isflexible and robust,and ii)the numerical solution is more accurate in the same magnitude of computational cost.As typical applications,we apply the new ap-proach to solve elliptic problems and(nonlinear)eigenvalue problems with essential boundary conditions,and numerical experiments are presented to show the effective-ness of the new method.

An Augmented Lagrangian Uzawa IterativeMethod for Solving Double Saddle-Point Systems with Semidefinite(2,2)Block and its Application to DLM/FDMethod for Elliptic Interface Problems

.In this paper,an augmented Lagrangian Uzawa iterative method is developed and analyzed for solving a class of double saddle-point systems with semidefinite(2,2)block.Convergence of the iterativemethod is proved under the assumption that the double saddle-point problem exists a unique solution.An application of the iterative method to the double saddle-point systems arising from the distributed Lagrange multiplier/fictitious domain(DLM/FD)finite element method for solving elliptic interface problems is also presented,in which the existence and uniqueness of the double saddle-point system is guaranteed by the analysis of the DLM/FD finite element method.Numerical experiments are conducted to validate the theoretical results and to study the performance of the proposed iterative method.

Multi-Material Topology Optimization for Spatial-Varying Porous Structures

This paper aims to propose a topology optimization method on generating porous structures comprising multiple materials.The mathematical optimization formulation is established under the constraints of individual volume fraction of constituent phase or total mass,as well as the local volume fraction of all phases.The original optimization problem with numerous constraints is converted into a box-constrained optimization problem by incorporating all constraints to the augmented Lagrangian function,avoiding the parameter dependence in the conventional aggregation process.Furthermore,the local volume percentage can be precisely satisfied.The effects including the globalmass bound,the influence radius and local volume percentage on final designs are exploited through numerical examples.The numerical results also reveal that porous structures keep a balance between the bulk design and periodic design in terms of the resulting compliance.All results,including those for irregular structures andmultiple volume fraction constraints,demonstrate that the proposedmethod can provide an efficient solution for multiple material infill structures.

A Fast Algorithm for Training Large Scale Support Vector Machines

The manuscript presents an augmented Lagrangian—fast projected gradient method (ALFPGM) with an improved scheme of working set selection, pWSS, a decomposition based algorithm for training support vector classification machines (SVM). The manuscript describes the ALFPGM algorithm, provides numerical results for training SVM on large data sets, and compares the training times of ALFPGM and Sequential Minimal Minimization algorithms (SMO) from Scikit-learn library. The numerical results demonstrate that ALFPGM with the improved working selection scheme is capable of training SVM with tens of thousands of training examples in a fraction of the training time of some widely adopted SVM tools.

IMAGE SUPER-RESOLUTION RECONSTRUCTION BY HUBER REGULARIZATION AND TAILORED FINITE POINT METHOD

In this paper,we propose using the tailored finite point method(TFPM)to solve the resulting parabolic or elliptic equations when minimizing the Huber regularization based image super-resolution model using the augmented Lagrangian method(ALM).The Hu-ber regularization based image super-resolution model can ameliorate the staircase for restored images.TFPM employs the method of weighted residuals with collocation tech-nique,which helps get more accurate approximate solutions to the equations and reserve more details in restored images.We compare the new schemes with the Marquina-Osher model,the image super-resolution convolutional neural network(SRCNN)and the classical interpolation methods:bilinear interpolation,nearest-neighbor interpolation and bicubic interpolation.Numerical experiments are presented to demonstrate that with the new schemes the quality of the super-resolution images has been improved.Besides these,the existence of the minimizer of the Huber regularization based image super-resolution model and the convergence of the proposed algorithm are also established in this paper.

Generalized Lagrangian Duality in Set-valued Vector Optimization via Abstract Subdifferential

In this paper,we investigate dual problems for nonconvex set-valued vector optimization via abstract subdifferential.We first introduce a generalized augmented Lagrangian function induced by a coupling vector-valued function for set-valued vector optimization problem and construct related set-valued dual map and dual optimization problem on the basic of weak efficiency,which used by the concepts of supremum and infimum of a set.We then establish the weak and strong duality results under this augmented Lagrangian and present sufficient conditions for exact penalization via an abstract subdifferential of the object map.Finally,we define the sub-optimal path related to the dual problem and show that every cluster point of this sub-optimal path is a primal optimal solution of the object optimization problem.In addition,we consider a generalized vector variational inequality as an application of abstract subdifferential.

Randomized Algorithms for Orthogonal Nonnegative Matrix Factorization

Orthogonal nonnegative matrix factorization(ONMF)is widely used in blind image separation problem,document classification,and human face recognition.The model of ONMF can be efficiently solved by the alternating direction method of multipliers and hierarchical alternating least squares method.When the given matrix is huge,the cost of computation and communication is too high.Therefore,ONMF becomes challenging in the large-scale setting.The random projection is an efficient method of dimensionality reduction.In this paper,we apply the random projection to ONMF and propose two randomized algorithms.Numerical experiments show that our proposed algorithms perform well on both simulated and real data.

Multi-area distributed three-phase state estimation for unbalanced active distribution networks

This paper proposes a new multi-area framework for unbalanced active distribution network(ADN) state estimation. Firstly, an innovative three-phase distributed generator(DG) model is presented to take the asymmetric characteristics of DG three-phase outputs into consideration. Then a feasible method to set pseudo-measurements for unmonitored DGs is introduced. The states of DGs,together with the states of alternating current(AC) buses in ADNs, were estimated by using the weighted least squares(WLS) method. After that, the ADN was divided into several independent subareas. Based on the augmented Lagrangian method, this work proposes a fully distributed three-phase state estimator for the multi-area ADN.Finally, from the simulation results on the modified IEEE123-bus system, the effectiveness and applicability of the proposed methodology have been investigated and discussed.