Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Exponential distribution-based genetic algorithm for solving mixed-integer bilevel programming problems

Two classes of mixed-integer nonlinear bilevel programming problems are discussed. One is that the follower＇s functions are separable with respect to the follower＇s variables, and the other is that the follower＇s functions are convex if the follower＇s variables are not restricted to integers. A genetic algorithm based on an exponential distribution is proposed for the aforementioned problems. First, for each fixed leader＇s variable x, it is proved that the optimal solution y of the follower＇s mixed-integer programming can be obtained by solving associated relaxed problems, and according to the convexity of the functions involved, a simplified branch and bound approach is given to solve the follower＇s programming for the second class of problems. Furthermore, based on an exponential distribution with a parameter λ, a new crossover operator is designed in which the best individuals are used to generate better offspring of crossover. The simulation results illustrate that the proposed algorithm is efficient and robust.

Computation of an Emptiable Minimal Siphon in a Subclass of Petri Nets Using Mixed-Integer Programming

Deadlock resolution strategies based on siphon control are widely investigated.Their computational efficiency largely depends on siphon computation.Mixed-integer programming(MIP)can be utilized for the computation of an emptiable siphon in a Petri net(PN).Based on it,deadlock resolution strategies can be designed without requiring complete siphon enumeration that has exponential complexity.Due to this reason,various MIP methods are proposed for various subclasses of PNs.This work proposes an innovative MIP method to compute an emptiable minimal siphon(EMS)for a subclass of PNs named S^(4)PR.In particular,many particular structural characteristics of EMS in S4 PR are formalized as constraints,which greatly reduces the solution space.Experimental results show that the proposed MIP method has higher computational efficiency.Furthermore,the proposed method allows one to determine the liveness of an ordinary S^(4)PR.

Improved genetic algorithm for nonlinear programming problems

An improved genetic algorithm（IGA） based on a novel selection strategy to handle nonlinear programming problems is proposed.Each individual in selection process is represented as a three-dimensional feature vector which is composed of objective function value,the degree of constraints violations and the number of constraints violations.It is easy to distinguish excellent individuals from general individuals by using an individuals＇ feature vector.Additionally,a local search（LS） process is incorporated into selection operation so as to find feasible solutions located in the neighboring areas of some infeasible solutions.The combination of IGA and LS should offer the advantage of both the quality of solutions and diversity of solutions.Experimental results over a set of benchmark problems demonstrate that IGA has better performance than other algorithms.

A DUAL-RELAX PENALTY FUNCTION APPROACH FOR SOLVING NONLINEAR BILEVEL PROGRAMMING WITH LINEAR LOWER LEVEL PROBLEM

The penalty function method, presented many years ago, is an important nu- merical method for the mathematical programming problems. In this article, we propose a dual-relax penalty function approach, which is significantly different from penalty func- tion approach existing for solving the bilevel programming, to solve the nonlinear bilevel programming with linear lower level problem. Our algorithm will redound to the error analysis for computing an approximate solution to the bilevel programming. The error estimate is obtained among the optimal objective function value of the dual-relax penalty problem and of the original bilevel programming problem. An example is illustrated to show the feasibility of the proposed approach.

NONLINEAR PROGRAMMING VIA AN EXACT PENALTY FUNCTION：CONVERGENCE RATE ANALYSIS

ＮＯＮＬＩＮＥＡＲＰＲＯＧＲＡＭＭＩＮＧＶＩＡＡＮＥＸＡＣＴＰＥＮＡＬＴＹＦＵＮＣＴＩＯＮ：ＣＯＮＶＥＲＧＥＮＣＥＲＡＴＥＡＮＡＬＹＳＩＳＬｉＸｕｅｑｕａｎＬｉＳｏｎｇｒｅｎＨａｎＸｕｉｌｉ（ＤｅｐａｒｔｍｅｎｔｏｆＡｐｐｌｉｅｄＭａｔｈｅｍａｔｉｃ...

Nonlinear Model-Based Process Operation under UncertaintyUsing Exact Parametric Programming

In the present work, two new, （multi-）parametric programming （mp-P）-inspired algorithms for the solutionof mixed-integer nonlinear programming （MINLP） problems are developed, with their main focus being onprocess synthesis problems. The algorithms are developed for the special case in which the nonlinearitiesarise because of logarithmic terms, with the first one being developed for the deterministic case, and thesecond for the parametric case （p-MINLP）. The key idea is to formulate and solve the square system of thefirst-order Karush-Kuhn-Tucker （KKT） conditions in an analytical way, by treating the binary variables and/or uncertain parameters as symbolic parameters. To this effect, symbolic manipulation and solution tech-niques are employed. In order to demonstrate the applicability and validity of the proposed algorithms, twoprocess synthesis case studies are examined. The corresponding solutions are then validated using state-of-the-art numerical MINLP solvers. For p-MINLP, the solution is given by an optimal solution as an explicitfunction of the uncertain parameters.

Further study on a class of augmented Lagrangians of Di Pillo and Grippo in nonlinear programming

In this paper, a class of augmented Lagrangiaus of Di Pillo and Grippo （DGALs） was considered, for solving equality-constrained problems via unconstrained minimization techniques. The relationship was further discussed between the uneonstrained minimizers of DGALs on the product space of problem variables and multipliers, and the solutions of the eonstrained problem and the corresponding values of the Lagrange multipliers. The resulting properties indicate more precisely that this class of DGALs is exact multiplier penalty functions. Therefore, a solution of the equslity-constralned problem and the corresponding values of the Lagrange multipliers can be found by performing a single unconstrained minimization of a DGAL on the product space of problem variables and multipliers.

A Combined Homotopy Infeasible Interior-Point Method for Convex Nonlinear Programming

In this paper, on the basis of the logarithmic barrier function and KKT conditions, we propose a combined homotopy infeasible interior-point method （CHIIP） for convex nonlinear programming problems. For any convex nonlinear programming, without strict convexity for the logarithmic barrier function, we get different solutions of the convex programming in different cases by CHIIP method.

Penalized interior point approach for constrained nonlinear programming

A penalized interior point approach for constrained nonlinear programming is examined in this work. To overcome the difficulty of initialization for the interior point method, a problem equivalent to the primal problem via incorporating an auxiliary variable is constructed. A combined approach of logarithm barrier and quadratic penalty function is proposed to solve the problem. Based on Newton＇s method, the global convergence of interior point and line search algorithm is proven. Only a finite number of iterations is required to reach an approximate optimal solution. Numerical tests are given to show the effectiveness of the method.

Modified Filled Function to Solve NonlinearProgramming Problem

Filled function method is an approach to find the global minimum of nonlinear functions. Many Problems, such as computing,communication control, and management, in real applications naturally result in global optimization formulations in a form ofnonlinear global integer programming. This paper gives a modified filled function method to solve the nonlinear global integerprogramming problem. The properties of the proposed modified filled function are also discussed in this paper. The results ofpreliminary numerical experiments are also reported.

NONLINEAR PROGRAMMING PROBLEMS IN MINE VENTILATION NETWORKS AND THEIR SOLUTIONS

Converting the balance equation of the branch of a mine ventilation network into an equivalent nonlinearprogramming problem,this paper proves that the total sum of the energy loss in every branch will be a minimumwhen the airflow distribution in the networks is in a balanced state.The energy means of solving the networkequations by nodal methods is also noted,and a theorem for the unique existence of the solution for a networkbalance equation is give.An example is used to explain these conclusions.

An Interval-parameter Fuzzy Robust Nonlinear Programming Model for Water Quality Management

Planning for water quality management is important for facilitating sustainable socio-economic development;however, the planning is also complicated by a variety of uncertainties and nonlinearities. In this study, an interval-parameter fuzzy robust nonlinear programming (IFRNP) model was developed for water quality management to deal with such difficulties. The developed model incorporated interval nonlinear programming (INP) and fuzzy robust programming (FRP) methods within a general optimization framework. The developed IFRNP model not only could explicitly deal with uncertainties represented as discrete interval numbers and fuzzy membership functions, but also was able to deal with nonlinearities in the objective function.

Novel Method to Handle Inequality Constraints for Nonlinear Programming

By redefining the multiplier associated with inequality constraint as a positive definite function of the originally-defined multiplier, say, u2_i, i=1, 2, ..., m, nonnegative constraints imposed on inequality constraints in Karush-Kuhn-Tucker necessary conditions are removed. For constructing the Lagrange neural network and Lagrange multiplier method, it is no longer necessary to convert inequality constraints into equality constraints by slack variables in order to reuse those results dedicated to equality constraints, and they can be similarly proved with minor modification. Utilizing this technique, a new type of Lagrange neural network and a new type of Lagrange multiplier method are devised, which both handle inequality constraints directly. Also, their stability and convergence are analyzed rigorously.

EXACT AUGMENTED LAGRANGIAN FUNCTION FOR NONLINEAR PROGRAMMING PROBLEMS WITH INEQUALITY CONSTRAINTS

An exact augmented Lagrangian function for the nonlinear nonconvex programming problems with inequality constraints was discussed. Under suitable hypotheses, the relationship was established between the local unconstrained minimizers of the augmented Lagrangian function on the space of problem variables and the local minimizers of the original constrained problem. Furthermore, under some assumptions, the relationship was also established between the global solutions of the augmented Lagrangian function on some compact subset of the space of problem variables and the global solutions of the constrained problem. Therefore, f^om the theoretical point of view, a solution of the inequality constrained problem and the corresponding values of the Lagrange multipliers can be found by the well-known method of multipliers which resort to the unconstrained minimization of the augmented Lagrangian function presented.

A Parametric Linearization Approach for Solving Zero-One Nonlinear Programming Problems

In this paper a new approach for obtaining an approximation global optimum solution of zero-one nonlinear programming (0-1 NP) problem which we call it Parametric Linearization Approach (P.L.A) is proposed. By using this approach the problem is transformed to a sequence of linear programming problems. The approximately solution of the original 0-1 NP problem is obtained based on the optimum values of the objective functions of this sequence of linear programming problems defined by (P.L.A).

A UNIVERSAL APPROACH FOR CONTINUOUS OR DISCRETE NONLINEAR PROGRAMMINGS WITH MULTIPLE VARIABLES AND CONSTRAINTS

A universal numerical approach for nonlinear mathematic programming problems is presented with an application of ratios of first-order differentials/differences of objective functions to constraint functions with respect to design variables. This approach can be efficiently used to solve continuous and, in particular, discrete programmings with arbitrary design variables and constraints. As a search method, this approach requires only computations of the functions and their partial derivatives or differences with respect to design variables, rather than any solution of mathematic equations. The present approach has been applied on many numerical examples as well as on some classical operational problems such as one-dimensional and two-dimensional knap-sack problems, one-dimensional and two-dimensional resource-distribution problems, problems of working reliability of composite systems and loading problems of machine, and more efficient and reliable solutions are obtained than traditional methods. The present approach can be used without limitation of modeling scales of the problem. Optimum solutions can be guaranteed as long as the objective function, constraint functions and their First-order derivatives/differences exist in the feasible domain or feasible set. There are no failures of convergence and instability when this approach is adopted.

Continuity of the optimal value function and optimal solutions of parametric mixed-integer quadratic programs

To properly describe and solve complex decision problems, research on theoretical properties and solution of mixed-integer quadratic programs is becoming very important. We establish in this paper different Lipschitz-type continuity results about the optimal value function and optimal solutions of mixed-integer parametric quadratic programs with parameters in the linear part of the objective function and in the right-hand sides of the linear constraints. The obtained results extend some existing results for continuous quadratic programs, and, more importantly, lay the foundation for further theoretical study and corresponding algorithm analysis on mixed-integer quadratic programs.

An Evolutionary Algorithm Based on a New Decomposition Scheme for Nonlinear Bilevel Programming Problems

In this paper, we focus on a class of nonlinear bilevel programming problems where the follower’s objective is a function of the linear expression of all variables, and the follower’s constraint functions are convex with respect to the follower’s variables. First, based on the features of the follower’s problem, we give a new decomposition scheme by which the follower’s optimal solution can be obtained easily. Then, to solve efficiently this class of problems by using evolutionary algorithm, novel evolutionary operators are designed by considering the best individuals and the diversity of individuals in the populations. Finally, based on these techniques, a new evolutionary algorithm is proposed. The numerical results on 20 test problems illustrate that the proposed algorithm is efficient and stable.