Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Max-plus-linear model-based predictive control for constrained hybrid systems: linear programming solution

In this paper, a linear programming method is proposed to solve model predictive control for a class of hybrid systems. Firstly, using the （max, ＋） algebra, a typical subclass of hybrid systems called max-plus-linear （MPL） systems is obtained. And then, model predictive control （MPC） framework is extended to MPL systems. In general, the nonlinear optimization approach or extended linear complementarity problem （ELCP） were applied to solve the MPL-MPC optimization problem. A new optimization method based on canonical forms for max-min-plus-scaling （MMPS） functions （using the operations maximization, minimization, addition and scalar multiplication） with linear constraints on the inputs is presented. The proposed approach consists in solving several linear programming problems and is more efficient than nonlinear optimization. The validity of the algorithm is illustrated by an example.

Safety control program for complex system based on behavior science

To control complex system's safety effectively,safety control program was supported based on the principles of behavioral science that shapes organizational be- havior,and organizational behavior produced individual behavior.The program can be structured into a model that consists of three modules including individual behavior rectifi- cation,organization behavior diagnosis and model of safety culture.The research result not only reveals the deep cause of complex system accidents but also provides structural descriptions with the accidents cause.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Optimal Control for a Class of Complex Singular System Based on Adaptive Dynamic Programming

This paper presents a new design approach to achieve decentralized optimal control of high-dimension complex singular systems with dynamic uncertainties. Based on robust adaptive dynamic programming(robust ADP) method, controllers for solving the singular systems optimal control problem are designed. The proposed algorithm can work well when the system model is not exactly known but the input and output data can be measured. The policy iteration of each controller only uses their own states and input information for learning,and do not need to know the whole system dynamics. Simulation results on the New England 10-machine 39-bus test system show the effectiveness of the designed controller.

Stability of Nonlinear Systems Using Optimal Fuzzy Controllers and Its Simulation by Java Programming

In this paper, at first, the single input rule modules(SIRMs) dynamically connected fuzzy inference model is used to stabilize a double inverted pendulum system. Then, a multiobjective particle swarm optimization(MOPSO) is implemented to optimize the fuzzy controller parameters in order to decrease the distance error of the cart and summation of the angle errors of the pendulums, simultaneously. The feasibility and efficiency of the proposed Pareto front is assessed in comparison with results reported in literature and obtained from other algorithms.Finally, the Java programming with applets is utilized to simulate the stability of the nonlinear system and explain the internetbased control.

Using Automation Controller System and Simulation Program for Testing Closed Circuits of Mini-Sprinkler Irrigation System

The field experiments were conducted at the experimental farm of Faculty of agricultural, southern Illinois University SIUC, USA. The project makes the irrigation automated. With the use of low cost sensors and the simple circuitry makes currently project a low cost product, which can be bought even by a poor farmer. This research work is best suited for places where water is scares and has to be used in limited quantity and this proposal is a model to modernize the agriculture industries at a mass scale with optimum expenditure. In the field of agricultural engineering, use of sensor method of irrigation operation is important and it is well known that closed circuits of Mini-sprinkler irrigation system are very economical and efficient. Closed circuits are considered one of the modifications of Mini-sprinkler irrigation system, and added advantages to Mini-sprinkler irrigation system because it can relieve low operating pressures problem at the end of the lateral lines. In the conventional closed circuits of Mini-sprinkler irrigation system, the farmer has to keep watch on irrigation timetable, which is different for different crops. Using this system, one can save manpower, water to improve production and ultimately profit. The data could be summarized in following: Irrigation methods under study when using lateral length 60 mcould be ranked in the following ascending order according the values of the predicted and measured head losses CM1M-SIS CM2M-SIS.The correlation (Corr.) coefficients were used to compare the predicted and measured head losses along the lateral lines of all the closed circuits designs. Generally, the values of correlation analysis were (>0.90) were obtained with 0% field slope60 mlength (experimental conditions) for all closed circuits.The interaction between irrigation methods: at the start there are significant differences between CM2M-SIS and CM1M-SIS.

Control strategy optimization using dynamic programming method for synergic electric system on hybrid electric vehicle

Dynamic Programming (DP) algorithm is used to find the optimal trajectories under Beijing cycle for the power management of synergic electric system (SES) which is composed of battery and super capacitor. Feasible rules are derived from analyzing the optimal trajectories, and it has the highest contribution to Hybrid Electric Vehicle (HEV). The methods of how to get the best performance is also educed. Using the new Rule-based power management strat-egy adopted from the optimal results, it is easy to demonstrate the effectiveness of the new strategy in further improvement of the fuel economy by the synergic hybrid system.

Digital close loop control system for current-programmed mode arc welding inverter

A small-signal model of current programmed mode pulse width modulation converter including the equivalent sampling effect is introduced and analyzed. In this model, an addition pole is brought out by the sampling effect in the current loop gain, and it affects dynamic bandwidth and stability of the inner current loop. By selecting the appropriate stability parameter which determines the additional pole and describes the degree of peaking in closed loop transfer function, a control model of current programmed full bridge arc welding inverter with maximum frequency bandwidth and stability can be obtained. Small and large amplitude pulse current outputs are employed in simulations and experiments and results validate the design method.

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems.Unlike existing optimal state feedback control,the control input of the optimal parallel control is introduced into the feedback system.However,due to the introduction of control input into the feedback system,the optimal state feedback control methods can not be applied directly.To address this problem,an augmented system and an augmented performance index function are proposed firstly.Thus,the general nonlinear system is transformed into an affine nonlinear system.The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically.It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function.Moreover,an adaptive dynamic programming(ADP)technique is utilized to implement the optimal parallel tracking control using a critic neural network(NN)to approximate the value function online.The stability analysis of the closed-loop system is performed using the Lyapunov theory,and the tracking error and NN weights errors are uniformly ultimately bounded(UUB).Also,the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals.Finally,the effectiveness of the developed optimal parallel control method is verified in two cases.

Dynamic Simulation of the Aircraft Environmental Control System

The dynamic research of aircraft environmental control system (ECS) is an important step in the advanced ECS design process. Based on the thermodynamics theory, mathematical models for the dynamic performance simulating of aircraft ECS were set up and an ECS simulation toolbox (ECS_1.0) was created with MATLAB language. It consists of main component modules (ducts, valves, heat exchangers, compressor, turbine, etc.). An aircraft environmental control system computer model was developed to assist engineers with the design and development of ECS dynamic optimization. An example simulating an existing ECS was given which shows the satisfactory effects.

Fuzzy neural network control of underwater vehicles based on desired state programming

Due to the nonlinearity and uncertainty, the precise control of underwater vehicles in some intelligent operations hasn’t been solved very well yet. A novel method of control based on desired state programming was presented, which used the technique of fuzzy neural network. The structure of fuzzy neural network was constructed according to the moving characters and the back propagation algorithm was deduced. Simulation experiments were conducted on general detection remotely operated vehicle. The results show that there is a great improvement in response and precision over traditional control, and good robustness to the model’s uncertainty and external disturbance, which has theoretical and practical value.

CONTROLLING ROBOT MANIPULATORS BY DYNAMIC PROGRAMMING

A certain number of considerations should be taken into account in the dynamic control of robot manipulators as highly complex non-linear systems.In this article,we provide a detailed presentation of the mechanical and electrical impli- cations of robots equipped with DC motor actuators.This model takes into account all non-linear aspects of the system.Then,we develop computational algorithms for optimal control based on dynamic programming.The robot's trajectory must be predefined,but performance criteria and constraints applying to the system are not limited and we may adapt them freely to the robot and the task being studied.As an example,a manipulator arm with 3 degrees of freedom is analyzed.

Pollution Control Programs of Large-Scale Livestock and Poultry Farms in the Dianchi Lake Basin

With the enlarging scale and intensifying production of livestock and poultry breeding, the environment pollution becomes increasingly prominent in the Dianchi Lake Basin since 1990s. According to the survey of ＂The First National Census of Pollution Sources＂, occurrence and discharge of pollutants in large-scale livestock and poultry farms in this region were first understood. The pollution characteristics of large-scale live- stock and poultry breeding were also analyzed deeply. On this basis, the significance of pollution control programs for environment protection was investigated from aspects of pollution control policy, technology management and publicity.

Tri-level programming model for combined urban traffic signal control and traffic flow guidance

In order to balance the temporal-spatial distribution of urban traffic flow, a model is established for combined urban traffic signal control and traffic flow guidance. With consideration of the wide use of fixed signal control at intersections, traffic assignment under traffic flow guidance, and dynamic characteristics of urban traffic management, a tri-level programming model is presented. To reflect the impact of intersection delay on traffic assignment, the lower level model is set as a modified user equilibrium model. The middle level model, which contains several definitional constraints for different phase modes, is built for the traffic signal control optimization. To solve the problem of tide lane management, the upper level model is built up based on nonlinear 0-1 integer programming. A heuristic iterative optimization algorithm(HIOA) is set up to solve the tri-level programming model. The lower level model is solved by method of successive averages(MSA), the middle level model is solved by non-dominated sorting genetic algorithm II(NSGA II), and the upper level model is solved by genetic algorithm(GA). A case study is raised to show the efficiency and applicability of the proposed modelling and computing method.

Application of Dynamic Programming Algorithm Based on Model Predictive Control in Hybrid Electric Vehicle Control Strategy

A good hybrid vehicle control strategy cannot only meet the power requirements of the vehicle,but also effectively save fuel and reduce emissions.In this paper,the construction of model predictive control in hybrid electric vehicle is proposed.The solving process and the use of reference trajectory are discussed for the application of MPC based on dynamic programming algorithm.The simulation of hybrid electric vehicle is carried out under a specific working condition.The simulation results show that the control strategy can effectively reduce fuel consumption when the torque of engine and motor is reasonably distributed,and the effectiveness of the control strategy is verified.

LabVIEW与Force Control组态软件DDE通讯的实现

文章阐述了利用DDE通讯技术,将Lab VIEW强大的数据处理能力、丰富的函数库资源与工业组态软件力控( Force Control)的易用性结合起来的方法,从而解决工业组态软件对复杂算法支持的不足,实现测控系统软件功能的扩展。

SEQUENTIAL QUADRATIC PROGRAMMING METHODS FOR OPTIMAL CONTROL PROBLEMS WITH STATE CONSTRAINTS

A kind of direct methods is presented for the solution of optimal control problems with state constraints. These methods are sequential quadratic programming methods. At every iteration a quadratic programming which is obtained by quadratic approximation to Lagrangian function and linear approximations to constraints is solved to get a search direction for a merit function. The merit function is formulated by augmenting the Lagrangian function with a penalty term. A line search is carried out along the search direction to determine a step length such that the merit function is decreased. The methods presented in this paper include continuous sequential quadratic programming methods and discreate sequential quadratic programming methods.

A Feasibility Study of an Individualized Voiding Program in Japan to Improve the Sense of Control in Older People with Functional Urinary Incontinence

Purpose: This study intended to assess the feasibility of an individualized voiding program in Japan aimed at improving the sense of micturition control in older people with functional urinary incontinence. Method: Following the interview guide, FGIs were conducted in two groups (4 - 6 participants) consisting of nurses and care workers with more than 5 years of experience as practitioners of urination care. Data were analyzed using a qualitative descriptive approach. Results: We determined that the program purpose is to “enable caregivers to work as a well-coordinated team to humanely facilitate excretion independence in older people, which is essential for living with dignity”, as this confirmed the importance of maintaining the sense of micturition control in older people for their well-being. In the program outline, we extracted the following five categories: 1) setting selection criteria for recipients considering the status of micturition induction, 2) careful collection of information and assessment of lower urinary tract symptoms in older people in case of environmental changes, 3) examination of methods used for assessing lower urinary tract symptoms according to the facility environment, 4) confirmation of the recipient’s micturition habits and request for assistance, and 5) conducting necessary examination for setting the intervention period and evaluation period according to the target condition. The feedback on the program guide was summarized in the statement—specific successful cases help frame and implement the micturition induction plan. Conclusion: We confirmed the feasibility of the micturition induction plan for improving the sense of control in older people with functional urinary incontinence. Upon evaluating the program guide, we deemed that referring to specific successful cases helps frame and implement the micturition induction plan. It is extremely important to verify the effectiveness of the program going forward.