Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Adaptive Trajectory Tracking Control for Nonholonomic Wheeled Mobile Robots:A Barrier Function Sliding Mode Approach

The trajectory tracking control performance of nonholonomic wheeled mobile robots(NWMRs)is subject to nonholonomic constraints,system uncertainties,and external disturbances.This paper proposes a barrier function-based adaptive sliding mode control(BFASMC)method to provide high-precision,fast-response performance and robustness for NWMRs.Compared with the conventional adaptive sliding mode control,the proposed control strategy can guarantee that the sliding mode variables converge to a predefined neighborhood of origin with a predefined reaching time independent of the prior knowledge of the uncertainties and disturbances bounds.Another advantage of the proposed algorithm is that the control gains can be adaptively adjusted to follow the disturbances amplitudes thanks to the barrier function.The benefit is that the overestimation of control gain can be eliminated,resulting in chattering reduction.Moreover,a modified barrier function-like control gain is employed to prevent the input saturation problem due to the physical limit of the actuator.The stability analysis and comparative experiments demonstrate that the proposed BFASMC can ensure the prespecified convergence performance of the NWMR system output variables and strong robustness against uncertainties/disturbances.

Quaternion-Based Adaptive Trajectory Tracking Control of a Rotor-Missile with Unknown Parameters Identification

This paper investigates the adaptive trajectory tracking control problem and the unknown parameter identification problem of a class of rotor-missiles with parametric system uncertainties.First,considering the uncertainty of structural and aerodynamic parameters,the six-degree-of-freedom(6Do F) nonlinear equations describing the position and attitude dynamics of the rotor-missile are established,respectively,in the inertial and body-fixed reference frames.Next,a hierarchical adaptive trajectory tracking controller that can guarantee closed-loop stability is proposed according to the cascade characteristics of the 6Do F dynamics.Then,a memory-augmented update rule of unknown parameters is proposed by integrating all historical data of the regression matrix.As long as the finitely excited condition is satisfied,the precise identification of unknown parameters can be achieved.Finally,the validity of the proposed trajectory tracking controller and the parameter identification method is proved through Lyapunov stability theory and numerical simulations.

Nonlinear robust adaptive control for bidirectional stabilization system of all-electric tank with unknown actuator backlash compensation and disturbance estimation

Since backlash nonlinearity is inevitably existing in actuators for bidirectional stabilization system of allelectric tank,it behaves more drastically in high maneuvering environments.In this work,the accurate tracking control for bidirectional stabilization system of moving all-electric tank with actuator backlash and unmodeled disturbance is solved.By utilizing the smooth adaptive backlash inverse model,a nonlinear robust adaptive feedback control scheme is presented.The unknown parameters and unmodelled disturbance are addressed separately through the derived parametric adaptive function and the continuous nonlinear robust term.Because the unknown backlash parameters are updated via adaptive function and the backlash effect can be suppressed successfully by inverse operation,which ensures the system stability.Meanwhile,the system disturbance in the high maneuverable environment can be estimated with the constructed adaptive law online improving the engineering practicality.Finally,Lyapunov-based analysis proves that the developed controller can ensure the tracking error asymptotically converges to zero even with unmodeled disturbance and unknown actuator backlash.Contrast co-simulations and experiments illustrate the advantages of the proposed approach.

Adaptive admittance tracking control for interactive robot with prescribed performance

An adaptive control approach is presented in this paper for tracking desired trajectories in interactive manipulators. The controller design incorporates prescribed performance functions (PPFs) to improve dynamic performance. Notably, the performance of the output error is confined in an envelope characterized by exponential convergence, leading to convergence to zero. This feature ensures a prompt response from admittance control and establishes a reliable safety framework for interactions. Simulation results provide practical insights,demonstrating the viability of the control scheme proposed in this paper.

A Fractional-Order Ultra-Local Model-Based Adaptive Neural Network Sliding Mode Control of n-DOF Upper-Limb Exoskeleton With Input Deadzone

This paper proposes an adaptive neural network sliding mode control based on fractional-order ultra-local model for n-DOF upper-limb exoskeleton in presence of uncertainties,external disturbances and input deadzone.Considering the model complexity and input deadzone,a fractional-order ultra-local model is proposed to formulate the original dynamic system for simple controller design.Firstly,the control gain of ultra-local model is considered as a constant.The fractional-order sliding mode technique is designed to stabilize the closed-loop system,while fractional-order time-delay estimation is combined with neural network to estimate the lumped disturbance.Correspondingly,a fractional-order ultra-local model-based neural network sliding mode controller(FO-NNSMC) is proposed.Secondly,to avoid disadvantageous effect of improper gain selection on the control performance,the control gain of ultra-local model is considered as an unknown parameter.Then,the Nussbaum technique is introduced into the FO-NNSMC to deal with the stability problem with unknown gain.Correspondingly,a fractional-order ultra-local model-based adaptive neural network sliding mode controller(FO-ANNSMC) is proposed.Moreover,the stability analysis of the closed-loop system with the proposed method is presented by using the Lyapunov theory.Finally,with the co-simulations on virtual prototype of 7-DOF iReHave upper-limb exoskeleton and experiments on 2-DOF upper-limb exoskeleton,the obtained compared results illustrate the effectiveness and superiority of the proposed method.

The prediction of projectile-target intersection for moving tank based on adaptive robust constraint-following control and interval uncertainty analysis

To improve the hit probability of tank at high speed,a prediction method of projectile-target intersection based on adaptive robust constraint-following control and interval uncertainty analysis is proposed.The method proposed provides a novel way to predict the impact point of projectile for moving tank.First,bidirectional stability constraints and stability constraint-following error are constructed using the Udwadia-Kalaba theory,and an adaptive robust constraint-following controller is designed considering uncertainties.Second,the exterior ballistic ordinary differential equation with uncertainties is integrated into the controller,and the pointing control of stability system is extended to the impact-point control of projectile.Third,based on the interval uncertainty analysis method combining Chebyshev polynomial expansion and affine arithmetic,a prediction method of projectile-target intersection is proposed.Finally,the co-simulation experiment is performed by establishing the multi-body system dynamic model of tank and mathematical model of control system.The results demonstrate that the prediction method of projectile-target intersection based on uncertainty analysis can effectively decrease the uncertainties of system,improve the prediction accuracy,and increase the hit probability.The adaptive robust constraint-following control can effectively restrain the uncertainties caused by road excitation and model error.

Reinforcement learning based adaptive control for uncertain mechanical systems with asymptotic tracking

This paper mainly focuses on the development of a learning-based controller for a class of uncertain mechanical systems modeled by the Euler-Lagrange formulation.The considered system can depict the behavior of a large class of engineering systems,such as vehicular systems,robot manipulators and satellites.All these systems are often characterized by highly nonlinear characteristics,heavy modeling uncertainties and unknown perturbations,therefore,accurate-model-based nonlinear control approaches become unavailable.Motivated by the challenge,a reinforcement learning(RL)adaptive control methodology based on the actor-critic framework is investigated to compensate the uncertain mechanical dynamics.The approximation inaccuracies caused by RL and the exogenous unknown disturbances are circumvented via a continuous robust integral of the sign of the error(RISE)control approach.Different from a classical RISE control law,a tanh(·)function is utilized instead of a sign(·)function to acquire a more smooth control signal.The developed controller requires very little prior knowledge of the dynamic model,is robust to unknown dynamics and exogenous disturbances,and can achieve asymptotic output tracking.Eventually,co-simulations through ADAMS and MATLAB/Simulink on a three degrees-of-freedom(3-DOF)manipulator and experiments on a real-time electromechanical servo system are performed to verify the performance of the proposed approach.

Adaptive Predefined-Time Backstepping Control for Grid Connected Photovoltaic Inverter

The system performance of grid-connected photovoltaic(PV)has a serious impact on the grid stability.To improve the control performance and shorten the convergence time,a predefined-time controller based on backstepping technology and dynamic surface control is formulated for the inverter in the grid-connected photovoltaic.The time-varying tuning functions are introduced into state-tracking errors to realize the predefined-time control effect.To address the“computational explosion problem”in the design process of backstepping control,dynamic surface control is adopted to avoid the analytical calculations of virtual control.The disturbances of the PV system are estimated and compensated by adaptive laws.The control parameters are chosen and the global stability of the closed-loop is ensured by Lyapunov conditions.Simulation results confirm the effectiveness of the proposed controller and ensure the predefined time control in the photovoltaic inverter.

Adaptive RISE Control of Winding Tension with Active Disturbance Rejection

A winding system is a time-varying system that considers complex nonlinear characteristics,and how to control the stability of the winding tension during the winding process is the primary problem that has hindered development in this field in recent years.Many nonlinear factors affect the tension in the winding process,such as friction,structured uncertainties,unstructured uncertainties,and external interference.These terms severely restrict the tension tracking performance.Existing tension control strategies are mainly based on the composite control of the tension and speed loops,and previous studies involve complex decoupling operations.Owing to the large number of calculations required for this method,it is inconvenient for practical engineering applications.To simplify the tension generation mechanism and the influence of the nonlinear characteristics of the winding system,a simpler nonlinear dynamic model of the winding tension was established.An adaptive method was applied to update the feedback gain of the continuous robust integral of the sign of the error(RISE).Furthermore,an extended state observer was used to estimate modeling errors and external disturbances.The model disturbance term can be compensated for in the designed RISE controller.The asymptotic stability of the system was proven according to the Lyapunov stability theory.Finally,a comparative analysis of the proposed nonlinear controller and several other controllers was performed.The results indicated that the control of the winding tension was significantly enhanced.

Adaptive Robust Servo Control for Vertical Electric Stabilization System of Tank and Experimental Validation

A tracking stability control problem for the vertical electric stabilization system of moving tank based on adaptive robust servo control is addressed.This paper mainly focuses on two types of possibly fast timevarying but bounded uncertainty within the vertical electric stabilization system:model parameter uncertainty and uncertain nonlinearity.First,the vertical electric stabilization system is constructed as an uncertain nonlinear dynamic system that can reflect the practical mechanics transfer process of the system.Second,the dynamical equation in the form of state space is established by designing the angular tracking error.Third,the comprehensive parameter of system uncertainty is designed to estimate the most conservative effects of uncertainty.Finally,an adaptive robust servo control which can effectively handle the combined effects of complex nonlinearity and uncertainty is proposed.The feasibility of the proposed control strategy under the practical physical condition is validated through the tests on the experimental platform.This paper pioneers the introduction of the internal nonlinearity and uncertainty of the vertical electric stabilization system into the settlement of the tracking stability control problem,and validates the advanced servo control strategy through experiment for the first time.

Adaptive Event-Triggered Time-Varying Output Group Formation Containment Control of Heterogeneous Multiagent Systems

In this paper,a class of time-varying output group formation containment control problem of general linear hetero-geneous multiagent systems(MASs)is investigated under directed topology.The MAS is composed of a number of tracking leaders,formation leaders and followers,where two different types of leaders are used to provide reference trajectories for movement and to achieve certain formations,respectively.Firstly,compen-sators are designed whose states are estimations of tracking lead-ers,based on which,a controller is developed for each formation leader to accomplish the expected formation.Secondly,two event-triggered compensators are proposed for each follower to evalu-ate the state and formation information of the formation leaders in the same group,respectively.Subsequently,a control protocol is designed for each follower,utilizing the output information,to guide the output towards the convex hull generated by the forma-tion leaders within the group.Next,the triggering sequence in this paper is decomposed into two sequences,and the inter-event intervals of these two triggering conditions are provided to rule out the Zeno behavior.Finally,a numerical simulation is intro-duced to confirm the validity of the proposed results.

Fuzzy Proportional Integral Derivative control of a voice coil actuator system for adaptive deformable mirrors

Research on adaptive deformable mirror technology for voice coil actuators(VCAs)is an important trend in the development of large ground-based telescopes.A voice coil adaptive deformable mirror contains a large number of actuators,and there are problems with structural coupling and large temperature increases in their internal coils.Additionally,parameters of the traditional proportional integral derivative(PID)control cannot be adjusted in real-time to adapt to system changes.These problems can be addressed by introducing fuzzy control methods.A table lookup method is adopted to replace real-time calculations of the regular fuzzy controller during the control process,and a prototype platform has been established to verify the effectiveness and robustness of this process.Experimental tests compare the control performance of traditional and fuzzy proportional integral derivative(Fuzzy-PID)controllers,showing that,in system step response tests,the fuzzy control system reduces rise time by 20.25%,decreases overshoot by 78.24%,and shortens settling time by 67.59%.In disturbance rejection experiments,fuzzy control achieves a 46.09%reduction in the maximum deviation,indicating stronger robustness.The Fuzzy-PID controller,based on table lookup,outperforms the standard controller significantly,showing excellent potential for enhancing the dynamic performance and disturbance rejection capability of the voice coil motor actuator system.

Identification of time-varying system and energy-based optimization of adaptive control in seismically excited structure

The combination of structural health monitoring and vibration control is of great importance to provide components of smart structures.While synthetic algorithms have been proposed,adaptive control that is compatible with changing conditions still needs to be used,and time-varying systems are required to be simultaneously estimated with the application of adaptive control.In this research,the identification of structural time-varying dynamic characteristics and optimized simple adaptive control are integrated.First,reduced variations of physical parameters are estimated online using the multiple forgetting factor recursive least squares(MFRLS)method.Then,the energy from the structural vibration is simultaneously specified to optimize the control force with the identified parameters to be operational.Optimization is also performed based on the probability density function of the energy under the seismic excitation at any time.Finally,the optimal control force is obtained by the simple adaptive control(SAC)algorithm and energy coefficient.A numerical example and benchmark structure are employed to investigate the efficiency of the proposed approach.The simulation results revealed the effectiveness of the integrated online identification and optimal adaptive control in systems.

A finite-time fuzzy adaptive output-feedback fault-tolerant control for underactuated wheeled mobile robots systems

This paper investigates the adaptive fuzzy finite-time output-feedback fault-tolerant control (FTC) problemfor a class of nonlinear underactuated wheeled mobile robots (UWMRs) system with intermittent actuatorfaults. The UWMR system includes unknown nonlinear dynamics and immeasurable states. Fuzzy logic systems(FLSs) are utilized to work out immeasurable functions. Furthermore, with the support of the backsteppingcontrol technique and adaptive fuzzy state observer, a fuzzy adaptive finite-time output-feedback FTC scheme isdeveloped under the intermittent actuator faults. It is testifying the scheme can ensure the controlled nonlinearUWMRs is stable and the estimation errors are convergent. Finally, the comparison results and simulationvalidate the effectiveness of the proposed fuzzy adaptive finite-time FTC approach.

Adaptive predictor-based control for a helicopter system with input delays:Design and experiments

In this paper,we consider a 2-degrees-of-freedom(DOF)helicopter system subject to long input delays and uncertain system parameters.To address the challenges including unknown system parameters and input delays in control design,we develop an adaptive predictor-feedback control law to achieve trajectory tracking.Stability of the closed-loop system is further established,where the tracking errors are shown to converge towards zero.Through simulation and experiments on the helicopter system,we illustrate that tracking of a desired trajectory is achieved with the proposed control scheme.

Implementation of Fuzzy Logic Control into an Equivalent Minimization Strategy for Adaptive Energy Management of A Parallel Hybrid Electric Vehicle

As government agencies continue to tighten emissions regulations due to the continued increase in greenhouse gas production, automotive industries are seeking to produce increasingly efficient vehicle technology. Hybrid electric vehicles (HEVs) have been introduced to mitigate problems while improving fuel economy. HEVs have led to the demand of creating more advanced controls software to consider multiple components for propulsive power in a vehicle. A large section in the software development process is the implementation of an optimal energy management strategy meant to improve the overall fuel efficiency of the vehicle. Optimal strategies can be implemented when driving conditions are known a prior. The Equivalent Consumption Minimization Strategy (ECMS) is an optimal control strategy that uses an equivalence factor to equate electrical to mechanical power when performing torque split determination between the internal combustion engine and electric motor for propulsive and regenerative torque. This equivalence factor is determined from offline vehicle simulations using a sensitivity analysis to provide optimal fuel economy results while maintaining predetermined high voltage battery state of charge (SOC) constraints. When the control hierarchy is modified or different driving styles are applied, the analysis must be redone to update the equivalence factor. The goal of this work is to implement a fuzzy logic controller that dynamically updates the equivalence factor to improve fuel economy, maintain a strict charge sustaining window of operation for the high voltage battery, and reduce computational time required during algorithm development. The adaptive algorithm is validated against global optimum fuel economy and charge sustaining results from a sensitivity analysis performed for multiple drive cycles. Results show a maximum fuel economy improvement of 9.82% when using a mild driving style and a 95% success rate when maintaining an ending SOC within 5% of the desired SOC regardless of starting SOC.

Novel Adaptive Memory Event-Triggered-Based Fuzzy Robust Control for Nonlinear Networked Systems via the Differential Evolution Algorithm

This article mainly investigates the fuzzy optimization robust control issue for nonlinear networked systems characterized by the interval type-2(IT2)fuzzy technique under a differential evolution algorithm.To provide a more reasonable utilization of the constrained communication channel,a novel adaptive memory event-triggered(AMET)mechanism is developed,where two event-triggered thresholds can be dynamically adjusted in the light of the current system information and the transmitted historical data.Sufficient conditions with less conservative design of the fuzzy imperfect premise matching(IPM)controller are presented by introducing the Wirtinger-based integral inequality,the information of membership functions(MFs)and slack matrices.Subsequently,under the IPM policy,a new MFs intelligent optimization technique that takes advantage of the differential evolution algorithm is first provided for IT2 TakagiSugeno(T-S)fuzzy systems to update the fuzzy controller MFs in real-time and achieve a better system control effect.Finally,simulation results demonstrate that the proposed control scheme can obtain better system performance in the case of using fewer communication resources.

An Adaptive Control Strategy for Energy Storage Interface Converter Based on Analogous Virtual Synchronous Generator

In the DC microgrid,the lack of inertia and damping in power electronic converters results in poor stability of DC bus voltage and low inertia of the DC microgrid during fluctuations in load and photovoltaic power.To address this issue,the application of a virtual synchronous generator(VSG)in grid-connected inverters control is referenced and proposes a control strategy called the analogous virtual synchronous generator(AVSG)control strategy for the interface DC/DC converter of the battery in the microgrid.Besides,a flexible parameter adaptive control method is introduced to further enhance the inertial behavior of the AVSG control.Firstly,a theoretical analysis is conducted on the various components of the DC microgrid,the structure of analogous virtual synchronous generator,and the control structure’s main parameters related to the DC microgrid’s inertial behavior.Secondly,the voltage change rate tracking coefficient is introduced to adjust the change of the virtual capacitance and damping coefficient flexibility,which further strengthens the inertia trend of the DC microgrid.Additionally,a small-signal modeling approach is used to analyze the approximate range of the AVSG’s main parameters ensuring system stability.Finally,conduct a simulation analysis by building the model of the DC microgrid system with photovoltaic(PV)and battery energy storage(BES)in MATLAB/Simulink.Simulation results from different scenarios have verified that the AVSG control introduces fixed inertia and damping into the droop control of the battery,resulting in a certain level of inertia enhancement.Furthermore,the additional adaptive control strategy built upon the AVSG control provides better and flexible inertial support for the DC microgrid,further enhances the stability of the DC bus voltage,and has a more positive impact on the battery performance.

Adaptive Robust Control with Leakage-Type Control Law for Trajectory Tracking of Exoskeleton Robots

This paper investigates the trajectory following problem of exoskeleton robots with numerous constraints. However, as a typical nonlinear system with variability and parameter uncertainty, it is difficult to accurately achieve the trajectory tracking control for exoskeletons. In this paper, we present a robust control of trajectory tracking control based on servo constraints. Firstly, we consider the uncertainties (e.g., modelling errors, initial condition deviations, structural vibrations, and other unknown external disturbances) in the exoskeleton system, which are time-varying and bounded. Secondly, we establish the dynamic model and formulate a close-loop connection between the dynamic model and the real world. Then, the trajectory tracking issue is regarded as a servo constraint problem, and an adaptive robust control with leakage-type adaptive law is proposed with the guaranteed Lyapunov stability. Finally, we conduct numerical simulations to verify the performance of the proposed controller.