Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Quaternion-Based Adaptive Trajectory Tracking Control of a Rotor-Missile with Unknown Parameters Identification

This paper investigates the adaptive trajectory tracking control problem and the unknown parameter identification problem of a class of rotor-missiles with parametric system uncertainties.First,considering the uncertainty of structural and aerodynamic parameters,the six-degree-of-freedom(6Do F) nonlinear equations describing the position and attitude dynamics of the rotor-missile are established,respectively,in the inertial and body-fixed reference frames.Next,a hierarchical adaptive trajectory tracking controller that can guarantee closed-loop stability is proposed according to the cascade characteristics of the 6Do F dynamics.Then,a memory-augmented update rule of unknown parameters is proposed by integrating all historical data of the regression matrix.As long as the finitely excited condition is satisfied,the precise identification of unknown parameters can be achieved.Finally,the validity of the proposed trajectory tracking controller and the parameter identification method is proved through Lyapunov stability theory and numerical simulations.

Nonlinear robust adaptive control for bidirectional stabilization system of all-electric tank with unknown actuator backlash compensation and disturbance estimation

Since backlash nonlinearity is inevitably existing in actuators for bidirectional stabilization system of allelectric tank,it behaves more drastically in high maneuvering environments.In this work,the accurate tracking control for bidirectional stabilization system of moving all-electric tank with actuator backlash and unmodeled disturbance is solved.By utilizing the smooth adaptive backlash inverse model,a nonlinear robust adaptive feedback control scheme is presented.The unknown parameters and unmodelled disturbance are addressed separately through the derived parametric adaptive function and the continuous nonlinear robust term.Because the unknown backlash parameters are updated via adaptive function and the backlash effect can be suppressed successfully by inverse operation,which ensures the system stability.Meanwhile,the system disturbance in the high maneuverable environment can be estimated with the constructed adaptive law online improving the engineering practicality.Finally,Lyapunov-based analysis proves that the developed controller can ensure the tracking error asymptotically converges to zero even with unmodeled disturbance and unknown actuator backlash.Contrast co-simulations and experiments illustrate the advantages of the proposed approach.

Adaptive admittance tracking control for interactive robot with prescribed performance

An adaptive control approach is presented in this paper for tracking desired trajectories in interactive manipulators. The controller design incorporates prescribed performance functions (PPFs) to improve dynamic performance. Notably, the performance of the output error is confined in an envelope characterized by exponential convergence, leading to convergence to zero. This feature ensures a prompt response from admittance control and establishes a reliable safety framework for interactions. Simulation results provide practical insights,demonstrating the viability of the control scheme proposed in this paper.

A Fractional-Order Ultra-Local Model-Based Adaptive Neural Network Sliding Mode Control of n-DOF Upper-Limb Exoskeleton With Input Deadzone

This paper proposes an adaptive neural network sliding mode control based on fractional-order ultra-local model for n-DOF upper-limb exoskeleton in presence of uncertainties,external disturbances and input deadzone.Considering the model complexity and input deadzone,a fractional-order ultra-local model is proposed to formulate the original dynamic system for simple controller design.Firstly,the control gain of ultra-local model is considered as a constant.The fractional-order sliding mode technique is designed to stabilize the closed-loop system,while fractional-order time-delay estimation is combined with neural network to estimate the lumped disturbance.Correspondingly,a fractional-order ultra-local model-based neural network sliding mode controller(FO-NNSMC) is proposed.Secondly,to avoid disadvantageous effect of improper gain selection on the control performance,the control gain of ultra-local model is considered as an unknown parameter.Then,the Nussbaum technique is introduced into the FO-NNSMC to deal with the stability problem with unknown gain.Correspondingly,a fractional-order ultra-local model-based adaptive neural network sliding mode controller(FO-ANNSMC) is proposed.Moreover,the stability analysis of the closed-loop system with the proposed method is presented by using the Lyapunov theory.Finally,with the co-simulations on virtual prototype of 7-DOF iReHave upper-limb exoskeleton and experiments on 2-DOF upper-limb exoskeleton,the obtained compared results illustrate the effectiveness and superiority of the proposed method.

The prediction of projectile-target intersection for moving tank based on adaptive robust constraint-following control and interval uncertainty analysis

To improve the hit probability of tank at high speed,a prediction method of projectile-target intersection based on adaptive robust constraint-following control and interval uncertainty analysis is proposed.The method proposed provides a novel way to predict the impact point of projectile for moving tank.First,bidirectional stability constraints and stability constraint-following error are constructed using the Udwadia-Kalaba theory,and an adaptive robust constraint-following controller is designed considering uncertainties.Second,the exterior ballistic ordinary differential equation with uncertainties is integrated into the controller,and the pointing control of stability system is extended to the impact-point control of projectile.Third,based on the interval uncertainty analysis method combining Chebyshev polynomial expansion and affine arithmetic,a prediction method of projectile-target intersection is proposed.Finally,the co-simulation experiment is performed by establishing the multi-body system dynamic model of tank and mathematical model of control system.The results demonstrate that the prediction method of projectile-target intersection based on uncertainty analysis can effectively decrease the uncertainties of system,improve the prediction accuracy,and increase the hit probability.The adaptive robust constraint-following control can effectively restrain the uncertainties caused by road excitation and model error.

Reinforcement learning based adaptive control for uncertain mechanical systems with asymptotic tracking

This paper mainly focuses on the development of a learning-based controller for a class of uncertain mechanical systems modeled by the Euler-Lagrange formulation.The considered system can depict the behavior of a large class of engineering systems,such as vehicular systems,robot manipulators and satellites.All these systems are often characterized by highly nonlinear characteristics,heavy modeling uncertainties and unknown perturbations,therefore,accurate-model-based nonlinear control approaches become unavailable.Motivated by the challenge,a reinforcement learning(RL)adaptive control methodology based on the actor-critic framework is investigated to compensate the uncertain mechanical dynamics.The approximation inaccuracies caused by RL and the exogenous unknown disturbances are circumvented via a continuous robust integral of the sign of the error(RISE)control approach.Different from a classical RISE control law,a tanh(·)function is utilized instead of a sign(·)function to acquire a more smooth control signal.The developed controller requires very little prior knowledge of the dynamic model,is robust to unknown dynamics and exogenous disturbances,and can achieve asymptotic output tracking.Eventually,co-simulations through ADAMS and MATLAB/Simulink on a three degrees-of-freedom(3-DOF)manipulator and experiments on a real-time electromechanical servo system are performed to verify the performance of the proposed approach.

Adaptive Robust Servo Control for Vertical Electric Stabilization System of Tank and Experimental Validation

A tracking stability control problem for the vertical electric stabilization system of moving tank based on adaptive robust servo control is addressed.This paper mainly focuses on two types of possibly fast timevarying but bounded uncertainty within the vertical electric stabilization system:model parameter uncertainty and uncertain nonlinearity.First,the vertical electric stabilization system is constructed as an uncertain nonlinear dynamic system that can reflect the practical mechanics transfer process of the system.Second,the dynamical equation in the form of state space is established by designing the angular tracking error.Third,the comprehensive parameter of system uncertainty is designed to estimate the most conservative effects of uncertainty.Finally,an adaptive robust servo control which can effectively handle the combined effects of complex nonlinearity and uncertainty is proposed.The feasibility of the proposed control strategy under the practical physical condition is validated through the tests on the experimental platform.This paper pioneers the introduction of the internal nonlinearity and uncertainty of the vertical electric stabilization system into the settlement of the tracking stability control problem,and validates the advanced servo control strategy through experiment for the first time.

Adaptive Event-Triggered Time-Varying Output Group Formation Containment Control of Heterogeneous Multiagent Systems

In this paper,a class of time-varying output group formation containment control problem of general linear hetero-geneous multiagent systems(MASs)is investigated under directed topology.The MAS is composed of a number of tracking leaders,formation leaders and followers,where two different types of leaders are used to provide reference trajectories for movement and to achieve certain formations,respectively.Firstly,compen-sators are designed whose states are estimations of tracking lead-ers,based on which,a controller is developed for each formation leader to accomplish the expected formation.Secondly,two event-triggered compensators are proposed for each follower to evalu-ate the state and formation information of the formation leaders in the same group,respectively.Subsequently,a control protocol is designed for each follower,utilizing the output information,to guide the output towards the convex hull generated by the forma-tion leaders within the group.Next,the triggering sequence in this paper is decomposed into two sequences,and the inter-event intervals of these two triggering conditions are provided to rule out the Zeno behavior.Finally,a numerical simulation is intro-duced to confirm the validity of the proposed results.

Adaptive Trajectory Tracking Control for Nonholonomic Wheeled Mobile Robots:A Barrier Function Sliding Mode Approach

The trajectory tracking control performance of nonholonomic wheeled mobile robots(NWMRs)is subject to nonholonomic constraints,system uncertainties,and external disturbances.This paper proposes a barrier function-based adaptive sliding mode control(BFASMC)method to provide high-precision,fast-response performance and robustness for NWMRs.Compared with the conventional adaptive sliding mode control,the proposed control strategy can guarantee that the sliding mode variables converge to a predefined neighborhood of origin with a predefined reaching time independent of the prior knowledge of the uncertainties and disturbances bounds.Another advantage of the proposed algorithm is that the control gains can be adaptively adjusted to follow the disturbances amplitudes thanks to the barrier function.The benefit is that the overestimation of control gain can be eliminated,resulting in chattering reduction.Moreover,a modified barrier function-like control gain is employed to prevent the input saturation problem due to the physical limit of the actuator.The stability analysis and comparative experiments demonstrate that the proposed BFASMC can ensure the prespecified convergence performance of the NWMR system output variables and strong robustness against uncertainties/disturbances.

Identification of time-varying system and energy-based optimization of adaptive control in seismically excited structure

The combination of structural health monitoring and vibration control is of great importance to provide components of smart structures.While synthetic algorithms have been proposed,adaptive control that is compatible with changing conditions still needs to be used,and time-varying systems are required to be simultaneously estimated with the application of adaptive control.In this research,the identification of structural time-varying dynamic characteristics and optimized simple adaptive control are integrated.First,reduced variations of physical parameters are estimated online using the multiple forgetting factor recursive least squares(MFRLS)method.Then,the energy from the structural vibration is simultaneously specified to optimize the control force with the identified parameters to be operational.Optimization is also performed based on the probability density function of the energy under the seismic excitation at any time.Finally,the optimal control force is obtained by the simple adaptive control(SAC)algorithm and energy coefficient.A numerical example and benchmark structure are employed to investigate the efficiency of the proposed approach.The simulation results revealed the effectiveness of the integrated online identification and optimal adaptive control in systems.

Adaptive predictor-based control for a helicopter system with input delays:Design and experiments

In this paper,we consider a 2-degrees-of-freedom(DOF)helicopter system subject to long input delays and uncertain system parameters.To address the challenges including unknown system parameters and input delays in control design,we develop an adaptive predictor-feedback control law to achieve trajectory tracking.Stability of the closed-loop system is further established,where the tracking errors are shown to converge towards zero.Through simulation and experiments on the helicopter system,we illustrate that tracking of a desired trajectory is achieved with the proposed control scheme.

Implementation of Fuzzy Logic Control into an Equivalent Minimization Strategy for Adaptive Energy Management of A Parallel Hybrid Electric Vehicle

As government agencies continue to tighten emissions regulations due to the continued increase in greenhouse gas production, automotive industries are seeking to produce increasingly efficient vehicle technology. Hybrid electric vehicles (HEVs) have been introduced to mitigate problems while improving fuel economy. HEVs have led to the demand of creating more advanced controls software to consider multiple components for propulsive power in a vehicle. A large section in the software development process is the implementation of an optimal energy management strategy meant to improve the overall fuel efficiency of the vehicle. Optimal strategies can be implemented when driving conditions are known a prior. The Equivalent Consumption Minimization Strategy (ECMS) is an optimal control strategy that uses an equivalence factor to equate electrical to mechanical power when performing torque split determination between the internal combustion engine and electric motor for propulsive and regenerative torque. This equivalence factor is determined from offline vehicle simulations using a sensitivity analysis to provide optimal fuel economy results while maintaining predetermined high voltage battery state of charge (SOC) constraints. When the control hierarchy is modified or different driving styles are applied, the analysis must be redone to update the equivalence factor. The goal of this work is to implement a fuzzy logic controller that dynamically updates the equivalence factor to improve fuel economy, maintain a strict charge sustaining window of operation for the high voltage battery, and reduce computational time required during algorithm development. The adaptive algorithm is validated against global optimum fuel economy and charge sustaining results from a sensitivity analysis performed for multiple drive cycles. Results show a maximum fuel economy improvement of 9.82% when using a mild driving style and a 95% success rate when maintaining an ending SOC within 5% of the desired SOC regardless of starting SOC.

An Adaptive Control Strategy for Energy Storage Interface Converter Based on Analogous Virtual Synchronous Generator

In the DC microgrid,the lack of inertia and damping in power electronic converters results in poor stability of DC bus voltage and low inertia of the DC microgrid during fluctuations in load and photovoltaic power.To address this issue,the application of a virtual synchronous generator(VSG)in grid-connected inverters control is referenced and proposes a control strategy called the analogous virtual synchronous generator(AVSG)control strategy for the interface DC/DC converter of the battery in the microgrid.Besides,a flexible parameter adaptive control method is introduced to further enhance the inertial behavior of the AVSG control.Firstly,a theoretical analysis is conducted on the various components of the DC microgrid,the structure of analogous virtual synchronous generator,and the control structure’s main parameters related to the DC microgrid’s inertial behavior.Secondly,the voltage change rate tracking coefficient is introduced to adjust the change of the virtual capacitance and damping coefficient flexibility,which further strengthens the inertia trend of the DC microgrid.Additionally,a small-signal modeling approach is used to analyze the approximate range of the AVSG’s main parameters ensuring system stability.Finally,conduct a simulation analysis by building the model of the DC microgrid system with photovoltaic(PV)and battery energy storage(BES)in MATLAB/Simulink.Simulation results from different scenarios have verified that the AVSG control introduces fixed inertia and damping into the droop control of the battery,resulting in a certain level of inertia enhancement.Furthermore,the additional adaptive control strategy built upon the AVSG control provides better and flexible inertial support for the DC microgrid,further enhances the stability of the DC bus voltage,and has a more positive impact on the battery performance.

Adaptive Optimal Output Regulation of Interconnected Singularly Perturbed Systems With Application to Power Systems

This article studies the adaptive optimal output regulation problem for a class of interconnected singularly perturbed systems(SPSs) with unknown dynamics based on reinforcement learning(RL).Taking into account the slow and fast characteristics among system states,the interconnected SPS is decomposed into the slow time-scale dynamics and the fast timescale dynamics through singular perturbation theory.For the fast time-scale dynamics with interconnections,we devise a decentralized optimal control strategy by selecting appropriate weight matrices in the cost function.For the slow time-scale dynamics with unknown system parameters,an off-policy RL algorithm with convergence guarantee is given to learn the optimal control strategy in terms of measurement data.By combining the slow and fast controllers,we establish the composite decentralized adaptive optimal output regulator,and rigorously analyze the stability and optimality of the closed-loop system.The proposed decomposition design not only bypasses the numerical stiffness but also alleviates the high-dimensionality.The efficacy of the proposed methodology is validated by a load-frequency control application of a two-area power system.

Adaptive Fixed-Time Control of Nonlinear MASs With Actuator Faults

The adaptive fixed-time consensus problem for a class of nonlinear multi-agent systems(MASs)with actuator faults is considered in this paper.To approximate the unknown nonlinear functions in MASs,radial basis function neural networks are used.In addition,the first order sliding mode differentiator is utilized to solve the“explosion of complexity”problem,and a filter error compensation method is proposed to ensure the convergence of filter error in fixed time.With the help of the Nussbaum function,the actuator failure compensation mechanism is constructed.By designing the adaptive fixed-time controller,all signals in MASs are bounded,and the consensus errors between the leader and all followers converge to a small area of origin.Finally,the effectiveness of the proposed control method is verified by simulation examples.

Modeling and Adaptive Neural Network Control for a Soft Robotic Arm With Prescribed Motion Constraints

This paper presents a dynamic model and performance constraint control of a line-driven soft robotic arm.The dynamics model of the soft robotic arm is established by combining the screw theory and the Cosserat theory.The unmodeled dynamics of the system are considered,and an adaptive neural network controller is designed using the backstepping method and radial basis function neural network.The stability of the closed-loop system and the boundedness of the tracking error are verified using Lyapunov theory.The simulation results show that our approach is a good solution to the motion constraint problem of the line-driven soft robotic arm.

System Identification and Parameter Self-Tuning Controller on Deep-Sea Mining Vehicle

System identification is a quintessential measure for real-time analysis on kinematic characteristics for deep-sea mining vehicle, and thus to enhance the control performance and testing efficiency. In this study, the system identification algorithm, recursive least square method with instrumental variables(IV-RLS), is tailored to model ‘Pioneer I’, a deep-sea mining vehicle which recently completed a 1305-meter-deep sea trial in the Xisha area of the South China Sea in August, 2021. The algorithm operates on the sensor data collected from the trial to obtain the vehicle’s kinematic model and accordingly design the parameter self-tuning controller. The performances demonstrate the accuracy of the model, and prove its generalization capability. With this model, the optimal controller has been designed, the control parameters have been self-tuned, and the response time and robustness of the system have been optimized,which validates the high efficiency on digital modelling for precision control of deep-sea mining vehicles.

Robust adaptive precision motion control of tank horizontal stabilizer based on unknown actuator backlash compensation

Backlash nonlinearity inevitably exists in the actuator of tank horizontal stabilizer and has adverse effect on the system control performance,however,how to effectively eliminate its effect remains a pending issue.To solve this problem,a robust adaptive precision motion controller is presented in this paper to address uncertainties and unknown actuator backlash of tank horizontal actuator.The controller handles the modeling uncertainties including parameter uncertainties and unmodeled disturbances by integrating adaptive feedforward compensation and continuous nonlinear robust law.Based on the backstepping method,a smooth backlash inverse model is constructed by combining the adaptive idea.Meanwhile,the unknown backlash parameters of the system can be approximated through the parameter adaptation,and the impact of the actuator backlash nonlinearity is effectively compensated via the inverse operation,which can availably improve the tracking performance.Moreover,the adaptive law can update the disturbance ranges of tank horizontal stabilizer online in real time,which enhances the feasibility in practical engineering applications.Furthermore,the stability analysis based on Lyapunov function shows that with the existence of unmodeled disturbances and unknown actuator backlash,the designed controller guarantees excellent asymptotic output tracking performance.Extensive comparative results verify the effectiveness of the proposed control strategy.

Hardware-in-loop adaptive neural control for a tiltable V-tail morphing aircraft

This paper proposes an adaptive neural control(ANC)method for the coupled nonlinear model of a novel type of embedded surface morphing aircraft which has a tiltable V-tail.A nonlinear model with sixdegrees-of-freedom is established.The first-order sliding mode differentiator(FSMD)is applied to the control scheme to avoid the problem of“differential explosion”.Radial basis function neural networks are introduced to estimate the uncertainty and external disturbance of the model,and an ANC controller is proposed based on this design idea.The stability of the proposed ANC controller is proved using Lyapunov theory,and the tracking error of the closed-loop system is semi-globally uniformly bounded.The effectiveness and robustness of the proposed method are verified by numerical simulations and hardware-in-the-loop(HIL)simulations.