Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Adaptive state-constrained/model-free iterative sliding mode control for aerial robot trajectory tracking

This paper develops a novel hierarchical control strategy for improving the trajectory tracking capability of aerial robots under parameter uncertainties.The hierarchical control strategy is composed of an adaptive sliding mode controller and a model-free iterative sliding mode controller(MFISMC).A position controller is designed based on adaptive sliding mode control(SMC)to safely drive the aerial robot and ensure fast state convergence under external disturbances.Additionally,the MFISMC acts as an attitude controller to estimate the unmodeled dynamics without detailed knowledge of aerial robots.Then,the adaption laws are derived with the Lyapunov theory to guarantee the asymptotic tracking of the system state.Finally,to demonstrate the performance and robustness of the proposed control strategy,numerical simulations are carried out,which are also compared with other conventional strategies,such as proportional-integralderivative(PID),backstepping(BS),and SMC.The simulation results indicate that the proposed hierarchical control strategy can fulfill zero steady-state error and achieve faster convergence compared with conventional strategies.

Reinforcement learning based adaptive control for uncertain mechanical systems with asymptotic tracking

This paper mainly focuses on the development of a learning-based controller for a class of uncertain mechanical systems modeled by the Euler-Lagrange formulation.The considered system can depict the behavior of a large class of engineering systems,such as vehicular systems,robot manipulators and satellites.All these systems are often characterized by highly nonlinear characteristics,heavy modeling uncertainties and unknown perturbations,therefore,accurate-model-based nonlinear control approaches become unavailable.Motivated by the challenge,a reinforcement learning(RL)adaptive control methodology based on the actor-critic framework is investigated to compensate the uncertain mechanical dynamics.The approximation inaccuracies caused by RL and the exogenous unknown disturbances are circumvented via a continuous robust integral of the sign of the error(RISE)control approach.Different from a classical RISE control law,a tanh(·)function is utilized instead of a sign(·)function to acquire a more smooth control signal.The developed controller requires very little prior knowledge of the dynamic model,is robust to unknown dynamics and exogenous disturbances,and can achieve asymptotic output tracking.Eventually,co-simulations through ADAMS and MATLAB/Simulink on a three degrees-of-freedom(3-DOF)manipulator and experiments on a real-time electromechanical servo system are performed to verify the performance of the proposed approach.

Sampled-data control through model-free reinforcement learning with effective experience replay

Reinforcement Learning(RL)based control algorithms can learn the control strategies for nonlinear and uncertain environment during interacting with it.Guided by the rewards generated by environment,a RL agent can learn the control strategy directly in a model-free way instead of investigating the dynamic model of the environment.In the paper,we propose the sampled-data RL control strategy to reduce the computational demand.In the sampled-data control strategy,the whole control system is of a hybrid structure,in which the plant is of continuous structure while the controller(RL agent)adopts a discrete structure.Given that the continuous states of the plant will be the input of the agent,the state–action value function is approximated by the fully connected feed-forward neural networks(FCFFNN).Instead of learning the controller at every step during the interaction with the environment,the learning and acting stages are decoupled to learn the control strategy more effectively through experience replay.In the acting stage,the most effective experience obtained during the interaction with the environment will be stored and during the learning stage,the stored experience will be replayed to customized times,which helps enhance the experience replay process.The effectiveness of proposed approach will be verified by simulation examples.

Adaptive Multi-Step Evaluation Design With Stability Guarantee for Discrete-Time Optimal Learning Control

This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman(HJB)equation.Then,the stability of the system is analyzed using control policies generated by MsHDP.Also,a general stability criterion is designed to determine the admissibility of the current control policy.That is,the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP.Further,based on the convergence and the stability criterion,the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly.Besides,actor-critic is utilized to implement the integrated MsHDP scheme,where neural networks are used to evaluate and improve the iterative policy as the parameter architecture.Finally,two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.

Model-Free Adaptive Frequency Calibration for Voltage-Controlled Oscillators

The operating frequency accuracy of the local oscillators is critical for the overall system performance in the communication systems.However,the high-precision oscillators could be too expensive for civil applications.In this paper,we propose a model-free adaptive frequency calibration framework for a voltage-controlled crystal oscillator(VCO)equipped with a time to digital converter(TDC),which can significantly improve the frequency accuracy of the VCO thus calibrated.The idea is to utilize a high-precision TDC to directly measure the VCO period which is then passed to a model-free method for working frequency calibration.One advantage of this method is that the working frequency calibration employs the system history of input/output(I/O)data,instead of establishing an accurate VCO voltagecontrolled oscillator model.Another advantage is the lightweight calibration method with low complexity such that it can be implemented on an MCU with limited computation capabilities.Experimental results show that the proposed calibration method can improve the frequency accuracy of a VCO from±20 ppm to±10 ppb,which indicates the promise of the modelfree adaptive frequency calibrator for VCOs.

Event-Triggered Asymmetric Bipartite Consensus Tracking for Nonlinear Multi-Agent Systems Based on Model-Free Adaptive Control

In this paper,an asymmetric bipartite consensus problem for the nonlinear multi-agent systems with cooperative and antagonistic interactions is studied under the event-triggered mechanism.For the agents described by a structurally balanced signed digraph,the asymmetric bipartite consensus objective is firstly defined,assigning the agents'output to different signs and module values.Considering with the completely unknown dynamics of the agents,a novel event-triggered model-free adaptive bipartite control protocol is designed based on the agents'triggered outputs and an equivalent compact form data model.By utilizing the Lyapunov analysis method,the threshold of the triggering condition is obtained.Subsequently,the asymptotic convergence of the tracking error is deduced and a sufficient condition is obtained based on the contraction mapping principle.Finally,the simulation example further demonstrates the effectiveness of the protocol.

Event-triggered model-free adaptive control for a class of surface vessels with time-delay and external disturbance via state observer

This paper provides an improved model-free adaptive control(IMFAC)strategy for solving the surface vessel trajectory tracking issue with time delay and restricted disturbance.Firstly,the original nonlinear time-delay system is transformed into a structure consisting of an unknown residual term and a parameter term with control inputs using a local compact form dynamic linearization(local-CFDL).To take advantage of the resulting structure,use a discrete-time extended state observer(DESO)to estimate the unknown residual factor.Then,according to the study,the inclusion of a time delay has no effect on the linearization structure,and an improved control approach is provided,in which DESO is used to adjust for uncertainties.Furthermore,a DESO-based event-triggered model-free adaptive control(ET-DESO-MFAC)is established by designing event-triggered conditions to assure Lyapunov stability.Only when the system’s indicator fulfills the provided event-triggered condition will the control input signal be updated;otherwise,the control input will stay the same as it is at the last trigger moment.A coordinate compensation approach is developed to reduce the steady-state inaccuracy of trajectory tracking.Finally,simulation experiments are used to assess the effectiveness of the proposed technique for trajectory tracking.

A Fractional-Order Ultra-Local Model-Based Adaptive Neural Network Sliding Mode Control of n-DOF Upper-Limb Exoskeleton With Input Deadzone

This paper proposes an adaptive neural network sliding mode control based on fractional-order ultra-local model for n-DOF upper-limb exoskeleton in presence of uncertainties,external disturbances and input deadzone.Considering the model complexity and input deadzone,a fractional-order ultra-local model is proposed to formulate the original dynamic system for simple controller design.Firstly,the control gain of ultra-local model is considered as a constant.The fractional-order sliding mode technique is designed to stabilize the closed-loop system,while fractional-order time-delay estimation is combined with neural network to estimate the lumped disturbance.Correspondingly,a fractional-order ultra-local model-based neural network sliding mode controller(FO-NNSMC) is proposed.Secondly,to avoid disadvantageous effect of improper gain selection on the control performance,the control gain of ultra-local model is considered as an unknown parameter.Then,the Nussbaum technique is introduced into the FO-NNSMC to deal with the stability problem with unknown gain.Correspondingly,a fractional-order ultra-local model-based adaptive neural network sliding mode controller(FO-ANNSMC) is proposed.Moreover,the stability analysis of the closed-loop system with the proposed method is presented by using the Lyapunov theory.Finally,with the co-simulations on virtual prototype of 7-DOF iReHave upper-limb exoskeleton and experiments on 2-DOF upper-limb exoskeleton,the obtained compared results illustrate the effectiveness and superiority of the proposed method.

Noise-Tolerant ZNN-Based Data-Driven Iterative Learning Control for Discrete Nonaffine Nonlinear MIMO Repetitive Systems

Aiming at the tracking problem of a class of discrete nonaffine nonlinear multi-input multi-output(MIMO) repetitive systems subjected to separable and nonseparable disturbances, a novel data-driven iterative learning control(ILC) scheme based on the zeroing neural networks(ZNNs) is proposed. First, the equivalent dynamic linearization data model is obtained by means of dynamic linearization technology, which exists theoretically in the iteration domain. Then, the iterative extended state observer(IESO) is developed to estimate the disturbance and the coupling between systems, and the decoupled dynamic linearization model is obtained for the purpose of controller synthesis. To solve the zero-seeking tracking problem with inherent tolerance of noise,an ILC based on noise-tolerant modified ZNN is proposed. The strict assumptions imposed on the initialization conditions of each iteration in the existing ILC methods can be absolutely removed with our method. In addition, theoretical analysis indicates that the modified ZNN can converge to the exact solution of the zero-seeking tracking problem. Finally, a generalized example and an application-oriented example are presented to verify the effectiveness and superiority of the proposed process.

Generalized projective synchronization of chaotic systems via adaptive learning control

In this paper, a learning control approach is applied to the generalized projective synchronisation （GPS） of different chaotic systems with unknown periodically time-varying parameters. Using the Lyapunov--Krasovskii functional stability theory, a differential-difference mixed parametric learning law and an adaptive learning control law are constructed to make the states of two different chaotic systems asymptotically synchronised. The scheme is successfully applied to the generalized projective synchronisation between the Lorenz system and Chen system. Moreover, numerical simulations results are used to verify the effectiveness of the proposed scheme.

Observer-based Adaptive Iterative Learning Control for Nonlinear Systems with Time-varying Delays

An observer-based adaptive iterative learning control （AILC） scheme is developed for a class of nonlinear systems with unknown time-varying parameters and unknown time-varying delays. The linear matrix inequality （LMI） method is employed to design the nonlinear observer. The designed controller contains a proportional-integral-derivative （PID） feedback term in time domain. The learning law of unknown constant parameter is differential-difference-type, and the learning law of unknown time-varying parameter is difference-type. It is assumed that the unknown delay-dependent uncertainty is nonlinearly parameterized. By constructing a Lyapunov-Krasovskii-like composite energy function （CEF）, we prove the boundedness of all closed-loop signals and the convergence of tracking error. A simulation example is provided to illustrate the effectiveness of the control algorithm proposed in this paper.

Adaptive Iterative Learning Control for Nonlinear Time-delay Systems with Periodic Disturbances Using FSE-neural Network

An adaptive iterative learning control scheme is presented for a class of strict-feedback nonlinear time-delay systems, with unknown nonlinearly parameterised and time-varying disturbed functions of known periods. Radial basis function neural network and Fourier series expansion （FSE） are combined into a new function approximator to model each suitable disturbed function in systems. The requirement of the traditional iterative learning control algorithm on the nonlinear functions （such as global Lipschitz condition） is relaxed. Furthermore, by using appropriate Lyapunov-Krasovskii functionals, all signs in the closed loop system are guaranteed to be semiglobally uniformly ultimately bounded, and the output of the system is proved to converge to the desired trajectory. A simulation example is provided to illustrate the effectiveness of the control scheme.

An Exploration on Adaptive Iterative Learning Control for a Class of Commensurate High-order Uncertain Nonlinear Fractional Order Systems

This paper explores the adaptive iterative learning control method in the control of fractional order systems for the first time. An adaptive iterative learning control(AILC) scheme is presented for a class of commensurate high-order uncertain nonlinear fractional order systems in the presence of disturbance.To facilitate the controller design, a sliding mode surface of tracking errors is designed by using sufficient conditions of linear fractional order systems. To relax the assumption of the identical initial condition in iterative learning control(ILC), a new boundary layer function is proposed by employing MittagLeffler function. The uncertainty in the system is compensated for by utilizing radial basis function neural network. Fractional order differential type updating laws and difference type learning law are designed to estimate unknown constant parameters and time-varying parameter, respectively. The hyperbolic tangent function and a convergent series sequence are used to design robust control term for neural network approximation error and bounded disturbance, simultaneously guaranteeing the learning convergence along iteration. The system output is proved to converge to a small neighborhood of the desired trajectory by constructing Lyapnov-like composite energy function(CEF)containing new integral type Lyapunov function, while keeping all the closed-loop signals bounded. Finally, a simulation example is presented to verify the effectiveness of the proposed approach.

Opti mal higher order learning adaptive control approach for a class of SISO nonlinear systems

In this paper, an optimal higher order learning adaptive control approach is developed for a class of SISO nonlinear systems. This design is model-free and depends directly on pseudo-partial-derivatives derived on-line from the input and output information of the system. A novel weighted one-step-ahead control criterion function is proposed for the control law. The convergence analysis shows that the proposed control law can guarantee the convergence under the assumption that the desired output is a set point. Simulation examples are provided for nonlinear systems to illustrate the better performance of the higher order learning adaptive control.

Adaptive Learning with Large Variability of Teaching Signals for Neural Networks and Its Application to Motion Control of an Industrial Robot

Recently, various control methods represented by proportional-integral-derivative （PID） control are used for robotic control. To cope with the requirements for high response and precision, advanced feedforward controllers such as gravity compensator, Coriolis/centrifugal force compensator and friction compensators have been built in the controller. Generally, it causes heavy computational load when calculating the compensating value within a short sampling period. In this paper, integrated recurrent neural networks are applied as a feedforward controller for PUMA560 manipulator. The feedforward controller works instead of gravity and Coriolis/centrifugal force compensators. In the learning process of the neural network by using back propagation algorithm, the learning coefficient and gain of sigmoid function are tuned intuitively and empirically according to teaching signals. The tuning is complicated because it is being conducted by trial and error. Especially, when the scale of teaching signal is large, the problem becomes crucial. To cope with the problem which concerns the learning performance, a simple and adaptive learning technique for large scale teaching signals is proposed. The learning techniques and control effectiveness are evaluated through simulations using the dynamic model of PUMA560 manipulator.

A Proposal of Adaptive PID Controller Based on Reinforcement Learning

Aimed at the lack of self-tuning PID parameters in conventional PID controllers, the structure and learning algorithm of an adaptive PID controller based on reinforcement learning were proposed. Actor-Critic learning was used to tune PID parameters in an adaptive way by taking advantage of the model-free and on-line learning properties of reinforcement learning effectively. In order to reduce the demand of storage space and to improve the learning efficiency, a single RBF neural network was used to approximate the policy function of Actor and the value function of Critic simultaneously. The inputs of RBF network are the system error, as well as the first and the second-order differences of error. The Actor can realize the mapping from the system state to PID parameters, while the Critic evaluates the outputs of the Actor and produces TD error. Based on TD error performance index and gradient descent method, the updating rules of RBF kernel function and network weights were given. Simulation results show that the proposed controller is efficient for complex nonlinear systems and it is perfectly adaptable and strongly robust, which is better than that of a conventional PID controller.

Adaptive learning tracking control of robotic manipulators with uncertainties

An adaptive learning tracking control scheme is developed for robotic manipulators by a synthesis of adaptive control and learning control approaches. The proposed controller possesses both adaptive and learning properties and thereby is able to handle robotic systems with both time-varying periodic uncertainties and time invariant parameters. Theoretical proofs are established to show that proposed controllers ensure asymptotical tracking performance. The effectiveness of the proposed approaches is validated through extensive numerical simulation results.

Model-free Adaptive Control for Spacecraft Attitude

A model-free adaptive control method is proposed for the spacecrafts whose dynamical parameters change over time and cannot be acquired accurately. The algorithm is based on full form dynamic linearization.A dimension reduction matrix is introduced to construct an augmented system with the same dimension input and output. The design of the controller depends on the system input and output data rather than the knowledge of the controlled plant. The numerical simulation results show that the improved controller can deal with different models with the same set of controller parameters,and the controller performance is better than that of PD controller for the time-varying system with disturbance.

Model-free adaptive control of space manipulator under different gravity environment

Due to the release of gravity in the space environment, the dynamic characteristics of the space manipulator have changed compared with that of the ground, which results in the change of its tracking precision. This paper presents a model-free adaptive control(MFAC) strategy to track the desired trajectory under different gravity environment. A dynamic transformation method and full form dynamic linearization(FFDL) approach are selected to dynamicly linearize the system, which can better eliminate the complex dynamics that may exist in the original system. The controlled object uses the two degrees of freedom of space manipulator and the controller only depends on the desired angle and torque of each joint of the space manipulator. Moreover, the proof of stability is also provided. Finally, simulation results are presented to demonstrate the effectiveness of the proposed strategy. It is shown that the proposed approach can achieve better trajectory tracking performance under different gravity environment without changing the control parameters, and the tracking precision can be significantly improved as compared with the proportional differential(PD) control results.