Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Controlling the dynamic behavior of decentralized cluster through centralized approaches

How to control the dynamic behavior of large-scale artificial active matter is a critical concern in experimental research on soft matter, particularly regarding the emergence of collective behaviors and the formation of group patterns. Centralized systems excel in precise control over individual behavior within a group, ensuring high accuracy and controllability in task execution. Nevertheless, their sensitivity to group size may limit their adaptability to diverse tasks. In contrast, decentralized systems empower individuals with autonomous decision-making, enhancing adaptability and system robustness. Yet, this flexibility comes at the cost of reduced accuracy and efficiency in task execution. In this work, we present a unique method for regulating the centralized dynamic behavior of self-organizing clusters based on environmental interactions. Within this environment-coupled robot system, each robot possesses similar dynamic characteristics, and their internal programs are entirely identical. However, their behaviors can be guided by the centralized control of the environment, facilitating the accomplishment of diverse cluster tasks. This approach aims to balance the accuracy and flexibility of centralized control with the robustness and task adaptability of decentralized control. The proactive regulation of dynamic behavioral characteristics in active matter groups, demonstrated in this work through environmental interactions, holds the potential to introduce a novel technological approach and provide experimental references for studying the dynamic behavior control of large-scale artificial active matter systems.

Dynamic Constraint-Driven Event-Triggered Control of Strict-Feedback Systems Without Max/Min Values on Irregular Constraints

This work proposes an event-triggered adaptive control approach for a class of uncertain nonlinear systems under irregular constraints.Unlike the constraints considered in most existing papers,here the external irregular constraints are considered and a constraints switching mechanism(CSM)is introduced to circumvent the difficulties arising from irregular output constraints.Based on the CSM,a new class of generalized barrier functions are constructed,which allows the control results to be independent of the maximum and minimum values(MMVs)of constraints and thus extends the existing results.Finally,we proposed a novel dynamic constraint-driven event-triggered strategy(DCDETS),under which the stress on signal transmission is reduced greatly and no constraints are violated by making a dynamic trade-off among system state,external constraints,and inter-execution intervals.It is proved that the system output is driven to close to the reference trajectory and the semi-global stability is guaranteed under the proposed control scheme,regardless of the external irregular output constraints.Simulation also verifies the effectiveness and benefits of the proposed method.

The Effect of Key Nodes on theMalware Dynamics in the Industrial Control Network

As industrialization and informatization becomemore deeply intertwined,industrial control networks have entered an era of intelligence.The connection between industrial control networks and the external internet is becoming increasingly close,which leads to frequent security accidents.This paper proposes a model for the industrial control network.It includes a malware containment strategy that integrates intrusion detection,quarantine,and monitoring.Basedonthismodel,the role of keynodes in the spreadofmalware is studied,a comparisonexperiment is conducted to validate the impact of the containment strategy.In addition,the dynamic behavior of the model is analyzed,the basic reproduction number is computed,and the disease-free and endemic equilibrium of the model is also obtained by the basic reproduction number.Moreover,through simulation experiments,the effectiveness of the containment strategy is validated,the influence of the relevant parameters is analyzed,and the containment strategy is optimized.In otherwords,selective immunity to key nodes can effectively suppress the spread ofmalware andmaintain the stability of industrial control systems.The earlier the immunization of key nodes,the better.Once the time exceeds the threshold,immunizing key nodes is almost ineffective.The analysis provides a better way to contain the malware in the industrial control network.

Hybrid Dynamic Variables-Dependent Event-Triggered Fuzzy Model Predictive Control

This article focuses on dynamic event-triggered mechanism(DETM)-based model predictive control(MPC) for T-S fuzzy systems.A hybrid dynamic variables-dependent DETM is carefully devised,which includes a multiplicative dynamic variable and an additive dynamic variable.The addressed DETM-based fuzzy MPC issue is described as a “min-max” optimization problem(OP).To facilitate the co-design of the MPC controller and the weighting matrix of the DETM,an auxiliary OP is proposed based on a new Lyapunov function and a new robust positive invariant(RPI) set that contain the membership functions and the hybrid dynamic variables.A dynamic event-triggered fuzzy MPC algorithm is developed accordingly,whose recursive feasibility is analysed by employing the RPI set.With the designed controller,the involved fuzzy system is ensured to be asymptotically stable.Two examples show that the new DETM and DETM-based MPC algorithm have the advantages of reducing resource consumption while yielding the anticipated performance.

Dynamic Event-Triggered Consensus Control for Input Constrained Multi-Agent Systems With a Designable Minimum Inter-Event Time

This paper investigates the consensus control of multi-agent systems(MASs) with constrained input using the dynamic event-triggered mechanism(ETM).Consider the MASs with small-scale networks where a centralized dynamic ETM with global information of the MASs is first designed.Then,a distributed dynamic ETM which only uses local information is developed for the MASs with large-scale networks.It is shown that the semi-global consensus of the MASs can be achieved by the designed bounded control protocol where the Zeno phenomenon is eliminated by a designable minimum inter-event time.In addition,it is easier to find a trade-off between the convergence rate and the minimum inter-event time by an adjustable parameter.Furthermore,the results are extended to regional consensus of the MASs with the bounded control protocol.Numerical simulations show the effectiveness of the proposed approach.

Practical bipartite consensus for multi-agent systems:A barrier function-based adaptive sliding-mode control approach

This paper is concerned with bipartite consensus tracking for multi-agent systems with unknown disturbances.A barrier function-based adaptive sliding-mode control(SMC)approach is proposed such that the bipartite steady-state error is converged to a predefined region of zero in finite time.Specifically,based on an error auxiliary taking neighboring antagonistic interactions into account,an SMC law is designed with an adaptive gain.The gain can switch to a positive semi-definite barrier function to ensure that the error auxiliary is constrained to a predefined neighborhood of zero,which in turn guarantees practical bipartite consensus tracking.A distinguished feature of the proposed controller is its independence on the bound of disturbances,while the input chattering phenomenon is alleviated.Finally,a numerical example is provided to verify the effectiveness of the proposed controller.

Transmission Dynamics and Optimal Control Strategies of a Hand-Foot-Mouth Disease Model with Treatment and Vaccination Interventions

In this article, the transmission dynamics of a Hand-Foot-Mouth disease model with treatment and vaccination interventions are studied. We calculated the basic reproduction number and proved the global stability of disease-free equilibrium when R0 R0 > 1. Meanwhile, we obtained the optimal control strategies minimizing the cost of intervention and minimizing the infected person. We also give some numerical simulations to verify our theoretical results.

Dynamic event-triggered formation control of second-order nonholonomic systems

In this paper,the formation control problem of secondorder nonholonomic mobile robot systems is investigated in a dynamic event-triggered scheme.Event-triggered control protocols combined with persistent excitation(PE)conditions are presented.In event-detecting processes,an inactive time is introduced after each sampling instant,which can ensure a positive minimum sampling interval.To increase the flexibility of the event-triggered scheme,internal dynamic variables are included in event-triggering conditions.Moreover,the dynamic event-triggered scheme plays an important role in increasing the lengths of time intervals between any two consecutive events.In addition,event-triggered control protocols without forward and angular velocities are also presented based on approximate-differentiation(low-pass)filters.The asymptotic convergence results are given based on a nested Matrosov theorem and artificial sampling methods.

Two-layer formation-containment fault-tolerant control of fixed-wing UAV swarm for dynamic target tracking

This paper tackles the formation-containment control problem of fixed-wing unmanned aerial vehicle(UAV)swarm with model uncertainties for dynamic target tracking in three-dimensional space in the faulty case of UAVs’actuator and sensor.The fixed-wing UAV swarm under consideration is organized as a“multi-leader-multi-follower”structure,in which only several leaders can obtain the dynamic target information while others only receive the neighbors’information through the communication network.To simultaneously realize the formation,containment,and dynamic target tracking,a two-layer control framework is adopted to decouple the problem into two subproblems:reference trajectory generation and trajectory tracking.In the upper layer,a distributed finite-time estimator(DFTE)is proposed to generate each UAV’s reference trajectory in accordance with the control objective.Subsequently,a distributed composite robust fault-tolerant trajectory tracking controller is developed in the lower layer,where a novel adaptive extended super-twisting(AESTW)algorithm with a finite-time extended state observer(FTESO)is involved in solving the robust trajectory tracking control problem under model uncertainties,actuator,and sensor faults.The proposed controller simultaneously guarantees rapidness and enhances the system’s robustness with fewer chattering effects.Finally,corresponding simulations are carried out to demonstrate the effectiveness and competitiveness of the proposed two-layer fault-tolerant cooperative control scheme.

Stability of Iterative Learning Control with Data Dropouts via Asynchronous Dynamical System

In this paper, the stability of iterative learning control with data dropouts is discussed. By the super vector formulation, an iterative learning control （ILC） system with data dropouts can be modeled as an asynchronous dynamical system with rate constraints on events in the iteration domain. The stability condition is provided in the form of linear matrix inequalities （LMIS） depending on the stability of asynchronous dynamical systems. The analysis is supported by simulations.

Vision-based Stabilization of Nonholonomic Mobile Robots by Integrating Sliding-mode Control and Adaptive Approach

Vision-based pose stabilization of nonholonomic mobile robots has received extensive attention. At present, most of the solutions of the problem do not take the robot dynamics into account in the controller design, so that these controllers are difficult to realize satisfactory control in practical application. Besides, many of the approaches suffer from the initial speed and torque jump which are not practical in the real world. Considering the kinematics and dynamics, a two-stage visual controller for solving the stabilization problem of a mobile robot is presented, applying the integration of adaptive control, sliding-mode control, and neural dynamics. In the first stage, an adaptive kinematic stabilization controller utilized to generate the command of velocity is developed based on Lyapunov theory. In the second stage, adopting the sliding-mode control approach, a dynamic controller with a variable speed function used to reduce the chattering is designed, which is utilized to generate the command of torque to make the actual velocity of the mobile robot asymptotically reach the desired velocity. Furthermore, to handle the speed and torque jump problems, the neural dynamics model is integrated into the above mentioned controllers. The stability of the proposed control system is analyzed by using Lyapunov theory. Finally, the simulation of the control law is implemented in perturbed case, and the results show that the control scheme can solve the stabilization problem effectively. The proposed control law can solve the speed and torque jump problems, overcome external disturbances, and provide a new solution for the vision-based stabilization of the mobile robot.

Distributed Secondary Control of AC Microgrids With External Disturbances and Directed Communication Topologies:A Full-Order Sliding-Mode Approach

This paper addresses the problem of distributed secondary control for islanded AC microgrids with external disturbances.By using a full-order sliding-mode(FOSM)approach,voltage regulation and frequency restoration are achieved in finite time.For voltage regulation,a distributed observer is proposed for each distributed generator(DG)to estimate a reference voltage level.Different from some conventional observers,the reference voltage level in this paper is accurately estimated under directed communication topologies.Based on the observer,a new nonlinear controller is designed in a backstepping manner such that an FOSM surface is reached in finite time.On the surface,the voltages of DGs are regulated to the reference level in finite time.For frequency restoration,a distributed controller is further proposed such that a constructed FOSM surface is reached in finite time,on which the frequencies of DGs are restored to a reference level in finite time under directed communication topologies.Finally,case studies on a modified IEEE 37-bus test system are conducted to demonstrate the effectiveness,the robustness against load changes,and the plug-and-play capability of the proposed controllers.

Adaptive Sliding-Mode Control of an Automotive Electronic Throttle in the Presence of Input Saturation Constraint

In modern vehicles, electronic throttle(ET) has been widely utilized to control the airflow into gasoline engine. To solve the control difficulties with an ET, such as strong nonlinearity,unknown model parameters and input saturation constraints,an adaptive sliding-mode tracking control strategy for an ET is presented. Compared with the existing control strategies for an ET, input saturation constraints and parameter uncertainties are adequately considered in the proposed control strategy. At first, the nonlinear dynamic model for control of an ET is described. According to the dynamical model, the nonlinear adaptive sliding-mode tracking control method is presented,where parameter adaptive laws and auxiliary design system are employed. Parameter adaptive law is given to estimate the unknown parameter with an ET. An auxiliary system is designed,and its state is utilized in the tracking control method to handle the input saturation. Stability proof and analysis of the adaptive sliding-mode control method is performed by using Lyapunov stability theory. Finally, the reliability and feasibility of the proposed control strategy are evaluated by computer simulation.Simulation research shows that the proposed sliding-mode control strategy can provide good control performance for an ET.

Adaptive functional link network control of near-space vehicles with dynamical uncertainties

The control law design for a near-space hypersonic vehicle（NHV） is highly challenging due to its inherent nonlinearity,plant uncertainties and sensitivity to disturbances.This paper presents a novel functional link network（FLN） control method for an NHV with dynamical thrust and parameter uncertainties.The approach devises a new partially-feedback-functional-link-network（PFFLN） adaptive law and combines it with the nonlinear generalized predictive control（NGPC） algorithm.The PFFLN is employed for approximating uncertainties in flight.Its weights are online tuned based on Lyapunov stability theorem for the first time.The learning process does not need any offline training phase.Additionally,a robust controller with an adaptive gain is designed to offset the approximation error.Finally,simulation results show a satisfactory performance for the NHV attitude tracking,and also illustrate the controller＇s robustness.

Mitochondrial dysfunction and quality control lie at the heart of subarachnoid hemorrhage

The dramatic increase in intracranial pressure after subarachnoid hemorrhage leads to a decrease in cerebral perfusion pressure and a reduction in cerebral blood flow.Mitochondria are directly affected by direct factors such as ischemia,hypoxia,excitotoxicity,and toxicity of free hemoglobin and its degradation products,which trigger mitochondrial dysfunction.Dysfunctional mitochondria release large amounts of reactive oxygen species,inflammatory mediators,and apoptotic proteins that activate apoptotic pathways,further damaging cells.In response to this array of damage,cells have adopted multiple mitochondrial quality control mechanisms through evolution,including mitochondrial protein quality control,mitochondrial dynamics,mitophagy,mitochondrial biogenesis,and intercellular mitochondrial transfer,to maintain mitochondrial homeostasis under pathological conditions.Specific interventions targeting mitochondrial quality control mechanisms have emerged as promising therapeutic strategies for subarachnoid hemorrhage.This review provides an overview of recent research advances in mitochondrial pathophysiological processes after subarachnoid hemorrhage,particularly mitochondrial quality control mechanisms.It also presents potential therapeutic strategies to target mitochondrial quality control in subarachnoid hemorrhage.

Adaptive sliding-mode path following control system of the underactuated USV under the influence of ocean currents

The path-following control of the asymmetry underactuated unmanned surface vehicle(USV) under external disturbances such as unknown constant and irrational ocean currents is discussed, and an adaptive sliding-mode path-following control system is proposed, which comprises a path-variable updated law,a modified integral line-of-sight(ILOS) guidance law based on a time-varying lookahead distance and adaptive feedback linearizing controllers combined with sliding-mode technique. A more accurate USV model without the assumption of having diagonal inertia and damping matrices is first presented, aiming at improving the performance of the path-following control. Next, the coordinate transformation is adopted to decouple the sway dynamic from the rudder angle, and the path-following errors dynamics without non-singular problem are presented in the moving Frenet-Serret frame. Then, based on the cascaded theorem and the adaptive sliding-mode method, the adaptive control law of position errors and course error are designed, among which the lookahead distance and integral gain are all computed as different functions of cross-track error to estimate and compensate the sideslip angle caused by external disturbances adaptively. Finally, according to the Lyapunov and cascaded theorem, the control system proposed is proved to be uniform globally asymptotic stability(UGAS) and uniform semiglobal exponential stability(USGES) when the control objectives are all achieved. Simulation results illustrate the precision and high-quality performance of this new controller.

Complete synchronization of uncertain chaotic dynamical network via a simple adaptive control

In this paper, based on the invaxiance principle of differential equations, we propose a simple adaptive control method to synchronize the network with coupling of the general form. Comparing with other control approaches, this scheme only depends on each node＇s state output. So we need not to know the concrete network structure and the solutions of the isolate nodes of the network in advance. In order to demonstrate the effectiveness of the method, a special example is provided and numerical simulations are performed. The numerical results show that our control scheme is very effective and robust against the weak noise.

SYNCHRONIZATION OF MASTER-SLAVE MARKOVIAN SWITCHING COMPLEX DYNAMICAL NETWORKS WITH TIME-VARYING DELAYS IN NONLINEAR FUNCTION VIA SLIDING MODE CONTROL

In this article, a synchronization problem for master-slave Markovian switching complex dynamical networks with time-varying delays in nonlinear function via sliding mode control is investigated. On the basis of the appropriate Lyapunov-Krasovskii functional, introducing some free weighting matrices, new synchronization criteria are derived in terms of linear matrix inequalities （LMIs）. Then, an integral sliding surface is designed to guarantee synchronization of master-slave Markovian switching complex dynamical networks, and the suitable controller is synthesized to ensure that the trajectory of the closed-loop error system can be driven onto the prescribed sliding mode surface. By using Dynkin＇s formula, we established the stochastic stablity of master-slave system. Finally, numerical example is provided to demonstrate the effectiveness of the obtained theoretical results.

Disturbance rejection and H_∞ pinning control of linear complex dynamical networks

This paper concerns the disturbance rejection problem of a linear complex dynamical network subject to external disturbances. A dynamical network is said to be robust to disturbance, if the H∞ norm of its transfer function matrix from the disturbance to the performance variable is satisfactorily small. It is shown that the disturbance rejection problem of a dynamical network can be solved by analysing the H∞ control problem of a set of independent systems whose dimensions are equal to that of a single node. A counter-intuitive result is that the disturbance rejection level of the whole network with a diffusive coupling will never be better than that of an isolated node. To improve this, local feedback injections are applied to a small fraction of the nodes in the network. Some criteria for possible performance improvement are derived in terms of linear matrix inequalities. It is further demonstrated via a simulation example that one can indeed improve the disturbance rejection level of the network by pinning the nodes with higher degrees than pinning those with lower degrees.