Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Design and Simulation of an Audio Signal Alerting and Automatic Control System

A large part of our daily lives is spent with audio information. Massive obstacles are frequently presented by the colossal amounts of acoustic information and the incredibly quick processing times. This results in the need for applications and methodologies that are capable of automatically analyzing these contents. These technologies can be applied in automatic contentanalysis and emergency response systems. Breaks in manual communication usually occur in emergencies leading to accidents and equipment damage. The audio signal does a good job by sending a signal underground, which warrants action from an emergency management team at the surface. This paper, therefore, seeks to design and simulate an audio signal alerting and automatic control system using Unity Pro XL to substitute manual communication of emergencies and manual control of equipment. Sound data were trained using the neural network technique of machine learning. The metrics used are Fast Fourier transform magnitude, zero crossing rate, root mean square, and percentage error. Sounds were detected with an error of approximately 17%;thus, the system can detect sounds with an accuracy of 83%. With more data training, the system can detect sounds with minimal or no error. The paper, therefore, has critical policy implications about communication, safety, and health for underground mine.

Development, Delivery and Evaluation of a Safety Training Program on Demolition for Blight Reduction

The main objective of the work presented in this paper was to develop a customized safety training program that can be incorporated into the demolition projects undertaken as part of blight reduction efforts in urban centers. A subsidiary objective was to devise and implement a safety program evaluation methodology, and gain insights on the relationships between knowledge acquisition through training and trainee demographics. Salient aspects of blight elimination efforts, as well as the main facets of building demolition practices and requirements, were reviewed. Information on various related safety and health hazards was studied in depth with a focus on demolition operations dealing with blighted properties. A unique safety hazard awareness training program was created for demolition workers, contractors and inspectors based on this research. In addition to devising a curriculum of relevant training topics along with traditional and online delivery systems to be employed, effectiveness evaluation instruments were formulated. Based on the limited data collected from the trainees it was concluded that the program was well-received by them and provided effective learning. It was also found that no statistically significant associations existed between the knowledge gain of the trainees, and either their experience level or union status, after taking this training.

Automatic Control of Automobile Transmission Performance Test Stand

To realize automatic control of automobile transmission performance test stand Methods The automatic control technique of the lubricant temperature,the program- controll- edautomaticshifting of the transmission,the continuous adjusting of revolution speed and load, data-acquisition and data real-time processing were adopted.Results The lubricant temperature was controlled at the set temperature ±2℃.The automatic shifting of the trans- mission is simple,reliable and accurate.The automatic adjusting of load and rotation speed is rapidandaccurate,the torque divergence is ±1N·m,the rotation speed divergence is ±5r/min Conclusion The four kinds of techniques are applied into the automobile transmission perfor- mance test stand successfully. mancetest stand successfully.

A Tentative Research on Complexity of Automatic Programming

In this paper, based on the following theoretical framework: Evolutionary Algorithms + Program Structures = Automatic Programming , some results on complexity of automatic programming for function modeling is given, which show that the complexity of automatic programming is an exponential function of the problem dimension N , the size of operator set ｜F｜ and the height of the program parse tree H . Following this results, the difficulties of automatic programming are discussed. Some function models discovered automatically from database by evolutionary modeling method are given, too.

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems.Unlike existing optimal state feedback control,the control input of the optimal parallel control is introduced into the feedback system.However,due to the introduction of control input into the feedback system,the optimal state feedback control methods can not be applied directly.To address this problem,an augmented system and an augmented performance index function are proposed firstly.Thus,the general nonlinear system is transformed into an affine nonlinear system.The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically.It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function.Moreover,an adaptive dynamic programming(ADP)technique is utilized to implement the optimal parallel tracking control using a critic neural network(NN)to approximate the value function online.The stability analysis of the closed-loop system is performed using the Lyapunov theory,and the tracking error and NN weights errors are uniformly ultimately bounded(UUB).Also,the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals.Finally,the effectiveness of the developed optimal parallel control method is verified in two cases.

Fuzzy neural network control of underwater vehicles based on desired state programming

Due to the nonlinearity and uncertainty, the precise control of underwater vehicles in some intelligent operations hasn’t been solved very well yet. A novel method of control based on desired state programming was presented, which used the technique of fuzzy neural network. The structure of fuzzy neural network was constructed according to the moving characters and the back propagation algorithm was deduced. Simulation experiments were conducted on general detection remotely operated vehicle. The results show that there is a great improvement in response and precision over traditional control, and good robustness to the model’s uncertainty and external disturbance, which has theoretical and practical value.

CONTROLLING ROBOT MANIPULATORS BY DYNAMIC PROGRAMMING

A certain number of considerations should be taken into account in the dynamic control of robot manipulators as highly complex non-linear systems.In this article,we provide a detailed presentation of the mechanical and electrical impli- cations of robots equipped with DC motor actuators.This model takes into account all non-linear aspects of the system.Then,we develop computational algorithms for optimal control based on dynamic programming.The robot's trajectory must be predefined,but performance criteria and constraints applying to the system are not limited and we may adapt them freely to the robot and the task being studied.As an example,a manipulator arm with 3 degrees of freedom is analyzed.

Pollution Control Programs of Large-Scale Livestock and Poultry Farms in the Dianchi Lake Basin

With the enlarging scale and intensifying production of livestock and poultry breeding, the environment pollution becomes increasingly prominent in the Dianchi Lake Basin since 1990s. According to the survey of ＂The First National Census of Pollution Sources＂, occurrence and discharge of pollutants in large-scale livestock and poultry farms in this region were first understood. The pollution characteristics of large-scale live- stock and poultry breeding were also analyzed deeply. On this basis, the significance of pollution control programs for environment protection was investigated from aspects of pollution control policy, technology management and publicity.

Tri-level programming model for combined urban traffic signal control and traffic flow guidance

In order to balance the temporal-spatial distribution of urban traffic flow, a model is established for combined urban traffic signal control and traffic flow guidance. With consideration of the wide use of fixed signal control at intersections, traffic assignment under traffic flow guidance, and dynamic characteristics of urban traffic management, a tri-level programming model is presented. To reflect the impact of intersection delay on traffic assignment, the lower level model is set as a modified user equilibrium model. The middle level model, which contains several definitional constraints for different phase modes, is built for the traffic signal control optimization. To solve the problem of tide lane management, the upper level model is built up based on nonlinear 0-1 integer programming. A heuristic iterative optimization algorithm(HIOA) is set up to solve the tri-level programming model. The lower level model is solved by method of successive averages(MSA), the middle level model is solved by non-dominated sorting genetic algorithm II(NSGA II), and the upper level model is solved by genetic algorithm(GA). A case study is raised to show the efficiency and applicability of the proposed modelling and computing method.

Design of a modern automatic control system for the activated sludge process in wastewater treatment

The Activated Sludge Process(ASP) exhibits highly nonlinear properties. The design of an automatic control system that is robust against disturbance of inlet wastewater flow rate and has short process settling times is a challenging matter. The proposed control method is an I-P modi fied controller automatic control system with state variable feedback and control canonical form simulation diagram for the process. A more stable response is achieved with this type of modern control. Settling times of 0.48 days are achieved for the concentration of microorganisms,(reference value step increase of 50 mg·L-1) and 0.01 days for the concentration of oxygen(reference value step increase of 0.1 mg·L-1). Fluctuations of concentrations of oxygen and microorganisms after an inlet disturbance of5 × 103m3·d-1are small. Changes in the reference values of oxygen and microorganisms(increases by 10%, 20% and 30%) show satisfactory response of the system in all cases. Changes in the value of inlet wastewater flow rate disturbance(increases by 10%, 25%, 50% and 100%) are stabilized by the control system in short time. Maximum percent overshoot is also taken in consideration in all cases and the largest value is 25% which is acceptable. The proposed method with I-P controller is better for disturbance rejection and process settling times compared to the same method using PI controller. This method can substitute optimal control systems in ASP.

Application of Dynamic Programming Algorithm Based on Model Predictive Control in Hybrid Electric Vehicle Control Strategy

A good hybrid vehicle control strategy cannot only meet the power requirements of the vehicle,but also effectively save fuel and reduce emissions.In this paper,the construction of model predictive control in hybrid electric vehicle is proposed.The solving process and the use of reference trajectory are discussed for the application of MPC based on dynamic programming algorithm.The simulation of hybrid electric vehicle is carried out under a specific working condition.The simulation results show that the control strategy can effectively reduce fuel consumption when the torque of engine and motor is reasonably distributed,and the effectiveness of the control strategy is verified.

Automatic differentiation for reduced sequential quadratic programming

In order to slove the large-scale nonlinear programming （NLP） problems efficiently, an efficient optimization algorithm based on reduced sequential quadratic programming （rSQP） and automatic differentiation （AD） is presented in this paper. With the characteristics of sparseness, relatively low degrees of freedom and equality constraints utilized, the nonlinear programming problem is solved by improved rSQP solver. In the solving process, AD technology is used to obtain accurate gradient information. The numerical results show that the combined algorithm, which is suitable for large-scale process optimization problems, can calculate more efficiently than rSQP itself.

Geometric model of roboticarc welding for automatic programming

Geometric information is important for automatic programming of arc welding robot. Complete geometric models of robotic arc welding are established in this paper. In the geometric model of weld seam, an equation with seam length as its parameter is introduced to represent any weld seam. The method to determine discrete programming points on a weld seam is presented. In the geometric model of weld workpiece, three class primitives and CSG tree are used to describe weld workpiece. Detailed data structure is presented. In pose transformation of torch, world frame, torch frame and active frame are defined, and transformation between frames is presented. Based on these geometric models, an automatic programming software package for robotic arc welding, RAWCAD, is developed. Experiments show that the geometric models are practical and reliable.

Design and realization of the automatic control system for the bell anneal furnace production

In accordance with the technical requirements of the full hydrogen bell anneal furnace(BAF)used for the annealing of stainless steel coils in the Baosteel Stainless Steel Business Unit,the design concept for the automatic control system was introduced in the paper.The major points described concerned the infrastructure of computers and networks,system interfaces,control functions,etc.The process control system of the full hydrogen BAF(L2)was the topic and its seven main functions were introduced.The expected output was achieved and manufacturing efficiency was enhanced within three months after the BAF units were put into operation,which demonstrated that the design concept of the automatic control system introduced in the paper was correct and that its functionalities were effective.

Max-plus-linear model-based predictive control for constrained hybrid systems: linear programming solution

In this paper, a linear programming method is proposed to solve model predictive control for a class of hybrid systems. Firstly, using the （max, ＋） algebra, a typical subclass of hybrid systems called max-plus-linear （MPL） systems is obtained. And then, model predictive control （MPC） framework is extended to MPL systems. In general, the nonlinear optimization approach or extended linear complementarity problem （ELCP） were applied to solve the MPL-MPC optimization problem. A new optimization method based on canonical forms for max-min-plus-scaling （MMPS） functions （using the operations maximization, minimization, addition and scalar multiplication） with linear constraints on the inputs is presented. The proposed approach consists in solving several linear programming problems and is more efficient than nonlinear optimization. The validity of the algorithm is illustrated by an example.

SEQUENTIAL QUADRATIC PROGRAMMING METHODS FOR OPTIMAL CONTROL PROBLEMS WITH STATE CONSTRAINTS

A kind of direct methods is presented for the solution of optimal control problems with state constraints. These methods are sequential quadratic programming methods. At every iteration a quadratic programming which is obtained by quadratic approximation to Lagrangian function and linear approximations to constraints is solved to get a search direction for a merit function. The merit function is formulated by augmenting the Lagrangian function with a penalty term. A line search is carried out along the search direction to determine a step length such that the merit function is decreased. The methods presented in this paper include continuous sequential quadratic programming methods and discreate sequential quadratic programming methods.

METARO^(3):Metamorphic Relation Group for Automatic Program Repair

The application of metamorphic testing(MT)on automatic program repair(APR-MT)is used to generate a patch without test oracles by examining whether the input metamorphic relation(MR)is satisfied or not.However,the delivered patch is plausible since it may satisfy the input MR but violate other MRs.This inspires us to propose an improved approach to enhance the effectiveness of APR-MT with metamorphic relation group.Ourapproach involves three major steps.First,we formally define the repair process of APR-MT by building the model of automatic program repair and metamorphic testing separately.Then,we propose the advanced model of automatic program repair based on metamorphic relation group,named METARO^(3),which takes several MRs as input while only one MR is used in APR-MT.We additionally present two kinds of selection strategies to rank MRs in descending order of the fault detection capability,which helps shorten the repair time of finding a patch.To demonstrate the feasibility and procedure of our approach,an illustration example was conducted.The results show that METARO^(3) can improve the effectiveness of APR-MT significantly.

Effectiveness of Automatic Control over Flash Distillation Kettle for Liquid-Phase Bulk Polymerization of Propylene

Automatic control over flash distillation section at SINOPEC Cangzhou Refinery has been realized, resulting in saving of propylene feedstock and emission reduction to harvest significant economic benefits.