Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems.Unlike existing optimal state feedback control,the control input of the optimal parallel control is introduced into the feedback system.However,due to the introduction of control input into the feedback system,the optimal state feedback control methods can not be applied directly.To address this problem,an augmented system and an augmented performance index function are proposed firstly.Thus,the general nonlinear system is transformed into an affine nonlinear system.The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically.It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function.Moreover,an adaptive dynamic programming(ADP)technique is utilized to implement the optimal parallel tracking control using a critic neural network(NN)to approximate the value function online.The stability analysis of the closed-loop system is performed using the Lyapunov theory,and the tracking error and NN weights errors are uniformly ultimately bounded(UUB).Also,the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals.Finally,the effectiveness of the developed optimal parallel control method is verified in two cases.

Optimal Control for a Class of Complex Singular System Based on Adaptive Dynamic Programming

This paper presents a new design approach to achieve decentralized optimal control of high-dimension complex singular systems with dynamic uncertainties. Based on robust adaptive dynamic programming(robust ADP) method, controllers for solving the singular systems optimal control problem are designed. The proposed algorithm can work well when the system model is not exactly known but the input and output data can be measured. The policy iteration of each controller only uses their own states and input information for learning,and do not need to know the whole system dynamics. Simulation results on the New England 10-machine 39-bus test system show the effectiveness of the designed controller.

Fuzzy neural network control of underwater vehicles based on desired state programming

Due to the nonlinearity and uncertainty, the precise control of underwater vehicles in some intelligent operations hasn’t been solved very well yet. A novel method of control based on desired state programming was presented, which used the technique of fuzzy neural network. The structure of fuzzy neural network was constructed according to the moving characters and the back propagation algorithm was deduced. Simulation experiments were conducted on general detection remotely operated vehicle. The results show that there is a great improvement in response and precision over traditional control, and good robustness to the model’s uncertainty and external disturbance, which has theoretical and practical value.

Program structures and synchronization schemes in numerical coordinate controllers of three-wire welding

In the three-wire welding system, a welding process consists of the operations of four devices, namely three welding machines and one bogie. The operations need to be synchronized by a numerical coordinate controller （ NCC ）. In this paper, we will discuss a tnsk-job-procedure cubic program structure. Under this structure, the devices are synchronized and isolated at the same time. This cubic program structure can also be used as a reference for other multi-device or multi-unit manufacturing processes.

CONTROLLING ROBOT MANIPULATORS BY DYNAMIC PROGRAMMING

A certain number of considerations should be taken into account in the dynamic control of robot manipulators as highly complex non-linear systems.In this article,we provide a detailed presentation of the mechanical and electrical impli- cations of robots equipped with DC motor actuators.This model takes into account all non-linear aspects of the system.Then,we develop computational algorithms for optimal control based on dynamic programming.The robot's trajectory must be predefined,but performance criteria and constraints applying to the system are not limited and we may adapt them freely to the robot and the task being studied.As an example,a manipulator arm with 3 degrees of freedom is analyzed.

Stability of Nonlinear Systems Using Optimal Fuzzy Controllers and Its Simulation by Java Programming

In this paper, at first, the single input rule modules(SIRMs) dynamically connected fuzzy inference model is used to stabilize a double inverted pendulum system. Then, a multiobjective particle swarm optimization(MOPSO) is implemented to optimize the fuzzy controller parameters in order to decrease the distance error of the cart and summation of the angle errors of the pendulums, simultaneously. The feasibility and efficiency of the proposed Pareto front is assessed in comparison with results reported in literature and obtained from other algorithms.Finally, the Java programming with applets is utilized to simulate the stability of the nonlinear system and explain the internetbased control.

Linear Programming for Optimum PID Controller Tuning

This work presents a new methodology based on Linear Programming (LP) to tune Proportional-Integral-Derivative (PID) control parameters. From a specification of a desired output time domain of the plant, a linear optimization system is proposed to adjust the PID controller leading the output signal to stable operation condition with minimum oscillations. The constraint set used in the optimization process is defined by using numerical integration approach. The generated optimization problem is convex and easily solved using an interior point algorithm. Results obtained using familiar plants from literature have shown that the proposed linear programming problem is very effective for tuning PID controllers.

Using Automation Controller System and Simulation Program for Testing Closed Circuits of Mini-Sprinkler Irrigation System

The field experiments were conducted at the experimental farm of Faculty of agricultural, southern Illinois University SIUC, USA. The project makes the irrigation automated. With the use of low cost sensors and the simple circuitry makes currently project a low cost product, which can be bought even by a poor farmer. This research work is best suited for places where water is scares and has to be used in limited quantity and this proposal is a model to modernize the agriculture industries at a mass scale with optimum expenditure. In the field of agricultural engineering, use of sensor method of irrigation operation is important and it is well known that closed circuits of Mini-sprinkler irrigation system are very economical and efficient. Closed circuits are considered one of the modifications of Mini-sprinkler irrigation system, and added advantages to Mini-sprinkler irrigation system because it can relieve low operating pressures problem at the end of the lateral lines. In the conventional closed circuits of Mini-sprinkler irrigation system, the farmer has to keep watch on irrigation timetable, which is different for different crops. Using this system, one can save manpower, water to improve production and ultimately profit. The data could be summarized in following: Irrigation methods under study when using lateral length 60 mcould be ranked in the following ascending order according the values of the predicted and measured head losses CM1M-SIS CM2M-SIS.The correlation (Corr.) coefficients were used to compare the predicted and measured head losses along the lateral lines of all the closed circuits designs. Generally, the values of correlation analysis were (>0.90) were obtained with 0% field slope60 mlength (experimental conditions) for all closed circuits.The interaction between irrigation methods: at the start there are significant differences between CM2M-SIS and CM1M-SIS.

Pollution Control Programs of Large-Scale Livestock and Poultry Farms in the Dianchi Lake Basin

With the enlarging scale and intensifying production of livestock and poultry breeding, the environment pollution becomes increasingly prominent in the Dianchi Lake Basin since 1990s. According to the survey of ＂The First National Census of Pollution Sources＂, occurrence and discharge of pollutants in large-scale livestock and poultry farms in this region were first understood. The pollution characteristics of large-scale live- stock and poultry breeding were also analyzed deeply. On this basis, the significance of pollution control programs for environment protection was investigated from aspects of pollution control policy, technology management and publicity.

Tri-level programming model for combined urban traffic signal control and traffic flow guidance

In order to balance the temporal-spatial distribution of urban traffic flow, a model is established for combined urban traffic signal control and traffic flow guidance. With consideration of the wide use of fixed signal control at intersections, traffic assignment under traffic flow guidance, and dynamic characteristics of urban traffic management, a tri-level programming model is presented. To reflect the impact of intersection delay on traffic assignment, the lower level model is set as a modified user equilibrium model. The middle level model, which contains several definitional constraints for different phase modes, is built for the traffic signal control optimization. To solve the problem of tide lane management, the upper level model is built up based on nonlinear 0-1 integer programming. A heuristic iterative optimization algorithm(HIOA) is set up to solve the tri-level programming model. The lower level model is solved by method of successive averages(MSA), the middle level model is solved by non-dominated sorting genetic algorithm II(NSGA II), and the upper level model is solved by genetic algorithm(GA). A case study is raised to show the efficiency and applicability of the proposed modelling and computing method.

Approximate Dynamic Programming for Self-Learning Control

This paper introduces a self-learning control approach based on approximate dynamic programming. Dynamic programming was introduced by Bellman in the 1950's for solving optimal control problems of nonlinear dynamical systems. Due to its high computational complexity, the applications of dynamic programming have been limited to simple and small problems. The key step in finding approximate solutions to dynamic programming is to estimate the performance index in dynamic programming. The optimal control signal can then be determined by minimizing (or maximizing) the performance index. Artificial neural networks are very efficient tools in representing the performance index in dynamic programming. This paper assumes the use of neural networks for estimating the performance index in dynamic programming and for generating optimal control signals, thus to achieve optimal control through self-learning.

Application of Dynamic Programming Algorithm Based on Model Predictive Control in Hybrid Electric Vehicle Control Strategy

A good hybrid vehicle control strategy cannot only meet the power requirements of the vehicle,but also effectively save fuel and reduce emissions.In this paper,the construction of model predictive control in hybrid electric vehicle is proposed.The solving process and the use of reference trajectory are discussed for the application of MPC based on dynamic programming algorithm.The simulation of hybrid electric vehicle is carried out under a specific working condition.The simulation results show that the control strategy can effectively reduce fuel consumption when the torque of engine and motor is reasonably distributed,and the effectiveness of the control strategy is verified.

Max-plus-linear model-based predictive control for constrained hybrid systems: linear programming solution

In this paper, a linear programming method is proposed to solve model predictive control for a class of hybrid systems. Firstly, using the （max, ＋） algebra, a typical subclass of hybrid systems called max-plus-linear （MPL） systems is obtained. And then, model predictive control （MPC） framework is extended to MPL systems. In general, the nonlinear optimization approach or extended linear complementarity problem （ELCP） were applied to solve the MPL-MPC optimization problem. A new optimization method based on canonical forms for max-min-plus-scaling （MMPS） functions （using the operations maximization, minimization, addition and scalar multiplication） with linear constraints on the inputs is presented. The proposed approach consists in solving several linear programming problems and is more efficient than nonlinear optimization. The validity of the algorithm is illustrated by an example.

Developing the Participatory Education Program for Dengue Prevention and Control in the Primary School, Southern Region, Thailand

Dengue disease had been the significant health problem of the primary school in southern Thailand. The objective was to develop the participatory program for dengue prevention and control in five primary schools. Participatory action research (PAR) was applied in five steps: 1) preparation, 2) assessment, 3) education program development, 4) implementation and 5) evaluation. Data collection used quantitative and qualitative methods in five primary schools in Southern, Thailand. Descriptive statistics and Chi-square (χ2) test were used to compare between pre- and post-intervention. Larval Indices were obtained from ratio analysis. The results showed the participatory program consisted of nine important activities for all stakeholders involving dengue problem. Health teacher’s activities and student’s activities were eight activities consisted of knowledge training, dengue manual, Computer Assisted Instructor (CAI), test, dengue data, larval indices survey, fish bank, and herbal garden. The seven activities of health provider were knowledge training, dengue manual, CAI, testing, dengue data, larval indices survey, and larval indices program. Three activities for students’ parents included dengue manual, dengue data, and larval indices survey. The 290 students were participating activities in education program and increasing basic knowledge of dengue prevention and control. Comparing the number of correct answers, >80% of students between pre- and post-intervention were found to increase basic knowledge from five correct answers to ten, and of these six items were significantly different (P < 0.001). The two primary schools were continuing non-positive containers (Container Index: CI = 0), but the total larval indices (Breteau Index: BI, House Index: HI, and Container Index: CI) of five primary schools showed high levels both pre- and post-intervention in primary schools;whereas, the students’ households showed high level of BI, HI and CI both pre- and post-intervention (754, 77, and 35 to 616, 81, and 32). Conclusion: The program was needed to continue in primary school and participate from health teacher, health provider, and students’ parents in order to be of great benefit for students.

SEQUENTIAL QUADRATIC PROGRAMMING METHODS FOR OPTIMAL CONTROL PROBLEMS WITH STATE CONSTRAINTS

A kind of direct methods is presented for the solution of optimal control problems with state constraints. These methods are sequential quadratic programming methods. At every iteration a quadratic programming which is obtained by quadratic approximation to Lagrangian function and linear approximations to constraints is solved to get a search direction for a merit function. The merit function is formulated by augmenting the Lagrangian function with a penalty term. A line search is carried out along the search direction to determine a step length such that the merit function is decreased. The methods presented in this paper include continuous sequential quadratic programming methods and discreate sequential quadratic programming methods.

Evaluation of a Voluntary Control Program for the Detection of Bovine Leukemia Virus Antibodies Based on Agar Gel Immunodiffusion Test in Dairy Farms in Costa Rica

Cattle from 20 dairy farms were serologically tested over a five-year period using agar gel immunodiffusion test (AGIDT) as part of a voluntary Bovine Leukemia Virus (BLV) control program. After five years of removing infected animals from the herds based on BLV-AGIDT serological status, blood samples from 332 cattle in these farms were collected and analyzed side by side by AGIDT and enzyme-linked immunosorbent assay (ELISA) to detect antibodies against BLV. AGIDT detected 29.2% (97) and 16.0% (53) of the animals as positive and weak positive respectively, whereas ELISA detected 58.2% (193) cattle as positive. The prevalence of BLV-antibodies determined with AGIDT in the dairy farms oscillated between 0% and 86%, whereas prevalence determined by ELISA ranged between 28% and 100% in the same farms. Although both techniques showed similarly results in farms with high BLV-prevalence, ELISA detected a larger proportion of BLV-positive, especially in farms with low or no BLV-prevalence based on AGIDT, leading to wrong assumptions in terms of farm level control efforts. Our results strongly suggest that AGIDT alone is inadequate to implement BLV control programs and ELISA is a more adequate test for BLV surveillance and control programs.

Control strategy optimization using dynamic programming method for synergic electric system on hybrid electric vehicle

Dynamic Programming (DP) algorithm is used to find the optimal trajectories under Beijing cycle for the power management of synergic electric system (SES) which is composed of battery and super capacitor. Feasible rules are derived from analyzing the optimal trajectories, and it has the highest contribution to Hybrid Electric Vehicle (HEV). The methods of how to get the best performance is also educed. Using the new Rule-based power management strat-egy adopted from the optimal results, it is easy to demonstrate the effectiveness of the new strategy in further improvement of the fuel economy by the synergic hybrid system.

A Feasibility Study of an Individualized Voiding Program in Japan to Improve the Sense of Control in Older People with Functional Urinary Incontinence

Purpose: This study intended to assess the feasibility of an individualized voiding program in Japan aimed at improving the sense of micturition control in older people with functional urinary incontinence. Method: Following the interview guide, FGIs were conducted in two groups (4 - 6 participants) consisting of nurses and care workers with more than 5 years of experience as practitioners of urination care. Data were analyzed using a qualitative descriptive approach. Results: We determined that the program purpose is to “enable caregivers to work as a well-coordinated team to humanely facilitate excretion independence in older people, which is essential for living with dignity”, as this confirmed the importance of maintaining the sense of micturition control in older people for their well-being. In the program outline, we extracted the following five categories: 1) setting selection criteria for recipients considering the status of micturition induction, 2) careful collection of information and assessment of lower urinary tract symptoms in older people in case of environmental changes, 3) examination of methods used for assessing lower urinary tract symptoms according to the facility environment, 4) confirmation of the recipient’s micturition habits and request for assistance, and 5) conducting necessary examination for setting the intervention period and evaluation period according to the target condition. The feedback on the program guide was summarized in the statement—specific successful cases help frame and implement the micturition induction plan. Conclusion: We confirmed the feasibility of the micturition induction plan for improving the sense of control in older people with functional urinary incontinence. Upon evaluating the program guide, we deemed that referring to specific successful cases helps frame and implement the micturition induction plan. It is extremely important to verify the effectiveness of the program going forward.