Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Controlling the dynamic behavior of decentralized cluster through centralized approaches

How to control the dynamic behavior of large-scale artificial active matter is a critical concern in experimental research on soft matter, particularly regarding the emergence of collective behaviors and the formation of group patterns. Centralized systems excel in precise control over individual behavior within a group, ensuring high accuracy and controllability in task execution. Nevertheless, their sensitivity to group size may limit their adaptability to diverse tasks. In contrast, decentralized systems empower individuals with autonomous decision-making, enhancing adaptability and system robustness. Yet, this flexibility comes at the cost of reduced accuracy and efficiency in task execution. In this work, we present a unique method for regulating the centralized dynamic behavior of self-organizing clusters based on environmental interactions. Within this environment-coupled robot system, each robot possesses similar dynamic characteristics, and their internal programs are entirely identical. However, their behaviors can be guided by the centralized control of the environment, facilitating the accomplishment of diverse cluster tasks. This approach aims to balance the accuracy and flexibility of centralized control with the robustness and task adaptability of decentralized control. The proactive regulation of dynamic behavioral characteristics in active matter groups, demonstrated in this work through environmental interactions, holds the potential to introduce a novel technological approach and provide experimental references for studying the dynamic behavior control of large-scale artificial active matter systems.

Dynamic Constraint-Driven Event-Triggered Control of Strict-Feedback Systems Without Max/Min Values on Irregular Constraints

This work proposes an event-triggered adaptive control approach for a class of uncertain nonlinear systems under irregular constraints.Unlike the constraints considered in most existing papers,here the external irregular constraints are considered and a constraints switching mechanism(CSM)is introduced to circumvent the difficulties arising from irregular output constraints.Based on the CSM,a new class of generalized barrier functions are constructed,which allows the control results to be independent of the maximum and minimum values(MMVs)of constraints and thus extends the existing results.Finally,we proposed a novel dynamic constraint-driven event-triggered strategy(DCDETS),under which the stress on signal transmission is reduced greatly and no constraints are violated by making a dynamic trade-off among system state,external constraints,and inter-execution intervals.It is proved that the system output is driven to close to the reference trajectory and the semi-global stability is guaranteed under the proposed control scheme,regardless of the external irregular output constraints.Simulation also verifies the effectiveness and benefits of the proposed method.

The Effect of Key Nodes on theMalware Dynamics in the Industrial Control Network

As industrialization and informatization becomemore deeply intertwined,industrial control networks have entered an era of intelligence.The connection between industrial control networks and the external internet is becoming increasingly close,which leads to frequent security accidents.This paper proposes a model for the industrial control network.It includes a malware containment strategy that integrates intrusion detection,quarantine,and monitoring.Basedonthismodel,the role of keynodes in the spreadofmalware is studied,a comparisonexperiment is conducted to validate the impact of the containment strategy.In addition,the dynamic behavior of the model is analyzed,the basic reproduction number is computed,and the disease-free and endemic equilibrium of the model is also obtained by the basic reproduction number.Moreover,through simulation experiments,the effectiveness of the containment strategy is validated,the influence of the relevant parameters is analyzed,and the containment strategy is optimized.In otherwords,selective immunity to key nodes can effectively suppress the spread ofmalware andmaintain the stability of industrial control systems.The earlier the immunization of key nodes,the better.Once the time exceeds the threshold,immunizing key nodes is almost ineffective.The analysis provides a better way to contain the malware in the industrial control network.

Hybrid Dynamic Variables-Dependent Event-Triggered Fuzzy Model Predictive Control

This article focuses on dynamic event-triggered mechanism(DETM)-based model predictive control(MPC) for T-S fuzzy systems.A hybrid dynamic variables-dependent DETM is carefully devised,which includes a multiplicative dynamic variable and an additive dynamic variable.The addressed DETM-based fuzzy MPC issue is described as a “min-max” optimization problem(OP).To facilitate the co-design of the MPC controller and the weighting matrix of the DETM,an auxiliary OP is proposed based on a new Lyapunov function and a new robust positive invariant(RPI) set that contain the membership functions and the hybrid dynamic variables.A dynamic event-triggered fuzzy MPC algorithm is developed accordingly,whose recursive feasibility is analysed by employing the RPI set.With the designed controller,the involved fuzzy system is ensured to be asymptotically stable.Two examples show that the new DETM and DETM-based MPC algorithm have the advantages of reducing resource consumption while yielding the anticipated performance.

Dynamic Event-Triggered Consensus Control for Input Constrained Multi-Agent Systems With a Designable Minimum Inter-Event Time

This paper investigates the consensus control of multi-agent systems(MASs) with constrained input using the dynamic event-triggered mechanism(ETM).Consider the MASs with small-scale networks where a centralized dynamic ETM with global information of the MASs is first designed.Then,a distributed dynamic ETM which only uses local information is developed for the MASs with large-scale networks.It is shown that the semi-global consensus of the MASs can be achieved by the designed bounded control protocol where the Zeno phenomenon is eliminated by a designable minimum inter-event time.In addition,it is easier to find a trade-off between the convergence rate and the minimum inter-event time by an adjustable parameter.Furthermore,the results are extended to regional consensus of the MASs with the bounded control protocol.Numerical simulations show the effectiveness of the proposed approach.

Transmission Dynamics and Optimal Control Strategies of a Hand-Foot-Mouth Disease Model with Treatment and Vaccination Interventions

In this article, the transmission dynamics of a Hand-Foot-Mouth disease model with treatment and vaccination interventions are studied. We calculated the basic reproduction number and proved the global stability of disease-free equilibrium when R0 R0 > 1. Meanwhile, we obtained the optimal control strategies minimizing the cost of intervention and minimizing the infected person. We also give some numerical simulations to verify our theoretical results.

Dynamic simulation analysis of molten salt reactor-coupled air-steam combined cycle power generation system

A nonlinear dynamic simulation model based on coordinated control of speed and flow rate for the molten salt reactor and combined cycle systems is proposed here to ensure the coordination and stability between the molten salt reactor and power system.This model considers the impact of thermal properties of fluid variation on accuracy and has been validated with Simulink.This study reveals the capability of the control system to compensate for anomalous situations and maintain shaft stability in the event of perturbations occurring in high-temperature molten salt tank outlet parameters.Meanwhile,the control system’s impact on the system’s dynamic characteristics under molten salt disturbance is also analyzed.The results reveal that after the disturbance occurs,the controlled system benefits from the action of the control,and the overshoot and disturbance amplitude are positively correlated,while the system power and frequency eventually return to the initial values.This simulation model provides a basis for utilizing molten salt reactors for power generation and maintaining grid stability.

An active high-static-low-dynamic-stiffness vibration isolator with adjustable buckling beams:theory and experiment

High-static-low-dynamic-stiffness(HSLDS)vibration isolators with buckling beams have been widely used to isolate external vibrations.An active adjustable device composed of proportion integration(PI)active controllers and piezoelectric actuators is proposed for improving the negative stiffness stroke of buckling beams.A nonlinear output frequency response function is used to analyze the effect of the vibration reduction.The prototype of the active HSLDS device is built,and the verification experiment is conducted.The results show that compared with the traditional HSLDS vibration isolator,the active HSLDS device can broaden the isolation frequency bandwidth,and effectively reduce the resonant amplitude by adjusting the active control parameters.The maximum vibration reduction rate of the active HSLDS vibration isolator can attain 89.9%,and the resonant frequency can be reduced from 31.08 Hz to 13.28 Hz.Therefore,this paper devotes to providing a new design scheme for enhanced HSLDS vibration isolators.

Dynamic event-triggered formation control of second-order nonholonomic systems

In this paper,the formation control problem of secondorder nonholonomic mobile robot systems is investigated in a dynamic event-triggered scheme.Event-triggered control protocols combined with persistent excitation(PE)conditions are presented.In event-detecting processes,an inactive time is introduced after each sampling instant,which can ensure a positive minimum sampling interval.To increase the flexibility of the event-triggered scheme,internal dynamic variables are included in event-triggering conditions.Moreover,the dynamic event-triggered scheme plays an important role in increasing the lengths of time intervals between any two consecutive events.In addition,event-triggered control protocols without forward and angular velocities are also presented based on approximate-differentiation(low-pass)filters.The asymptotic convergence results are given based on a nested Matrosov theorem and artificial sampling methods.

Two-layer formation-containment fault-tolerant control of fixed-wing UAV swarm for dynamic target tracking

This paper tackles the formation-containment control problem of fixed-wing unmanned aerial vehicle(UAV)swarm with model uncertainties for dynamic target tracking in three-dimensional space in the faulty case of UAVs’actuator and sensor.The fixed-wing UAV swarm under consideration is organized as a“multi-leader-multi-follower”structure,in which only several leaders can obtain the dynamic target information while others only receive the neighbors’information through the communication network.To simultaneously realize the formation,containment,and dynamic target tracking,a two-layer control framework is adopted to decouple the problem into two subproblems:reference trajectory generation and trajectory tracking.In the upper layer,a distributed finite-time estimator(DFTE)is proposed to generate each UAV’s reference trajectory in accordance with the control objective.Subsequently,a distributed composite robust fault-tolerant trajectory tracking controller is developed in the lower layer,where a novel adaptive extended super-twisting(AESTW)algorithm with a finite-time extended state observer(FTESO)is involved in solving the robust trajectory tracking control problem under model uncertainties,actuator,and sensor faults.The proposed controller simultaneously guarantees rapidness and enhances the system’s robustness with fewer chattering effects.Finally,corresponding simulations are carried out to demonstrate the effectiveness and competitiveness of the proposed two-layer fault-tolerant cooperative control scheme.

Robust missile autopilot design based on dynamic surface control

Since the dynamical system and control system of the missile are typically nonlinear, an effective acceleration tracking autopilot is designed using the dynamic surface control(DSC)technique in order to make the missile control system more robust despite the uncertainty of the dynamical parameters and the presence of disturbances. Firstly, the nonlinear mathematical model of the tail-controlled missile is decomposed into slow acceleration dynamics and fast pitch rate dynamics based on the naturally existing time scale separation. Secondly, the controller based on DSC is designed after obtaining the linear dynamics characteristics of the slow and fast subsystems. An extended state observer is used to detect the uncertainty of the system state variables and aerodynamic parameters to achieve the compensation of the control law. The closed-loop stability of the controller is derived and rigorously analyzed. Finally, the effectiveness and robustness of the design is verified by Monte Carlo simulation considering different initial conditions and parameter uptake. Simulation results illustrate that the missile autopilot based DSC controller achieves better performance and robustness than the other two well-known autopilots.The method proposed in this paper is applied to the design of a missile autopilot, and the results show that the acceleration tracking autopilot based on the DSC controller can ensure accurate tracking of the required commands and has better performance.

Fixed-Time Stabilization of a Class of Strict-Feedback Nonlinear Systems via Dynamic Gain Feedback Control

This paper presents a novel fixed-time stabilization control(FSC)method for a class of strict-feedback nonlinear systems involving unmodelled system dynamics.The key feature of the proposed method is the design of two dynamic parameters.Specifically,a set of auxiliary variables is first introduced through state transformation.These variables combine the original system states and the two introduced dynamic parameters,facilitating the closed-loop system stability analyses.Then,the two dynamic parameters are delicately designed by utilizing the Lyapunov method,ensuring that all the closed-loop system states are globally fixed-time stable.Compared with existing results,the“explosion of complexity”problem of backstepping control is avoided.Moreover,the two designed dynamic parameters are dependent on system states rather than a time-varying function,thus the proposed controller is still valid beyond the given fixedtime convergence instant.The effectiveness of the proposed method is demonstrated through two practical systems.

Dynamic Event-Triggered Fixed-Time Consensus Control and Its Applications to Magnetic Map Construction

This article deals with the consensus problem of multi-agent systems by developing a fixed-time consensus control approach with a dynamic event-triggered rule. First, a new fixedtime stability condition is obtained where the less conservative settling time is given such that the theoretical settling time can well reflect the real consensus time. Second, a dynamic event-triggered rule is designed to decrease the use of chip and network resources where Zeno behaviors can be avoided after consensus is achieved, especially for finite/fixed-time consensus control approaches. Third, in terms of the developed dynamic event-triggered rule, a fixed-time consensus control approach by introducing a new item is proposed to coordinate the multi-agent system to reach consensus. The corresponding stability of the multi-agent system with the proposed control approach and dynamic eventtriggered rule is analyzed based on Lyapunov theory and the fixed-time stability theorem. At last, the effectiveness of the dynamic event-triggered fixed-time consensus control approach is verified by simulations and experiments for the problem of magnetic map construction based on multiple mobile robots.

Experimental investigation of dynamic stall flow control using a microsecond-pulsed plasma actuator

To alleviate the performance deterioration caused by dynamic stall of a wind turbine airfoil,the flow control by a microsecond-pulsed dielectric barrier discharge(MP-DBD) actuator on the dynamic stall of a periodically pitching NACA0012 airfoil was investigated experimentally.Unsteady pressure measurements with high temporal accuracy were employed in this study,and the unsteady characteristics of the boundary layer were investigated by wavelet packet analysis and the moving root mean square method based on the acquired pressure.The experimental Mach number was 0.2,and the chord-based Reynolds number was 870 000.The dimensionless actuation frequencies F+ were chosen to be 0.5,1,2,and 3,respectively.For the light dynamic regime,the MP-DBD plasma actuator plays the role of suppressing flow separation from the trial edge and accelerating the flow reattachment due to the high-momentum freestream flow being entrained into the boundary layer.Meanwhile,actuation effects were promoted with the increasing dimensionless actuation frequency F+.The control effects of the deep dynamic stall were to delay the onset and reduce the strength of the dynamic stall vortex due to the accumulating vorticity near the leading edge being removed by the induced coherent vortex structures.The laminar fluctuation and Kelvin-Helmholtz(K-H) instabilities of transition and relaminarization were also mitigated by the MP-DBD actuation,and the alleviated K-H rolls led to the delay of the transition onset and earlier laminar reattachment,which improved the hysteresis effect of the dynamic stall.For the controlled cases of F+=2,and F+=3,the laminar fluctuation was replaced by relatively low frequency band disturbances corresponding to the harmonic responses of the MP-DBD actuation frequency.

Research on Mixed Logic Dynamic Modeling and Finite Control Set Model Predictive Control of Multi-Inverter Parallel System

Parallel connection of multiple inverters is an important means to solve the expansion,reserve and protection of distributed power generation,such as photovoltaics.In view of the shortcomings of traditional droop control methods such as weak anti-interference ability,low tracking accuracy of inverter output voltage and serious circulation phenomenon,a finite control set model predictive control(FCS-MPC)strategy of microgrid multiinverter parallel system based on Mixed Logical Dynamical(MLD)modeling is proposed.Firstly,the MLD modeling method is introduced logical variables,combining discrete events and continuous events to form an overall differential equation,which makes the modeling more accurate.Then a predictive controller is designed based on the model,and constraints are added to the objective function,which can not only solve the real-time changes of the control system by online optimization,but also effectively obtain a higher tracking accuracy of the inverter output voltage and lower total harmonic distortion rate(Total Harmonics Distortion,THD);and suppress the circulating current between the inverters,to obtain a good dynamic response.Finally,the simulation is carried out onMATLAB/Simulink to verify the correctness of the model and the rationality of the proposed strategy.This paper aims to provide guidance for the design and optimal control of multi-inverter parallel systems.

Coupled Dynamics and Integrated Control for Position and Attitude Motions of Spacecraft:A Survey

Inspired by the integrated guidance and control design for endo-atmospheric aircraft,the integrated position and attitude control of spacecraft has attracted increasing attention and gradually induced a wide variety of study results in last over two decades,fully incorporating control requirements and actuator characteristics of space missions.This paper presents a novel and comprehensive survey to the coupled position and attitude motions of spacecraft from the perspective of dynamics and control.To this end,a systematic analysis is firstly conducted in details to show the position and attitude mutual couplings of spacecraft.Particularly,in terms of the time discrepancy between spacecraft position and attitude motions,space missions can be categorized into two types:space proximity operation and space orbital maneuver.Based on this classification,the studies on the coupled dynamic modeling and the integrated control design for position and attitude motions of spacecraft are sequentially summarized and analyzed.On the one hand,various coupled position and dynamic formulations of spacecraft based on various mathematical tools are reviewed and compared from five aspects,including mission applicability,modeling simplicity,physical clearance,information matching and expansibility.On the other hand,the development of the integrated position and attitude control of spacecraft is analyzed for two space missions,and especially,five distinctive development trends are captured for space operation missions.Finally,insightful prospects on future development of the integrated position and attitude control technology of spacecraft are proposed,pointing out current primary technical issues and possible feasible solutions.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Functional Sustainability of a Flight Dynamics Control System for Stable Hovering Flight of an Unmanned Aerial Vehicle(UAV),such as in Agricultural Applications:Mathematical Modeling and Simulation

The simulation of a control system for the longitudinal axis of the rotary or fixed-wing unmanned aerial vehicles(UAVs)is demonstrated in this study.The control unit includes design considerations of two controllers to provide robust stability,tracking of the proposed linear dynamics,an adequate set of proportional-integral-derivative(PID)controller gains,and a minimal cost function.The PID control and linear quadratic regulator(LQR)with or without full-state-observer were evaluated.An optimal control system is assumed to provide fast rise and settling time,minimize overshoot,and eliminate the steady-state error.The effectiveness of this approach was verified by a linear model of the UAV aircraft in the semi-dynamic simulation platform of Matlab/Simulink,in which the open-loop system was assessed in terms of flight robustness and reference tracking.The experimental results show that the proposed controllers effectively improve the configuration of the control system of the plant,maintain the sustainability of the dynamic flight model stability,and diminish the flight controller errors.The LQR provides robust stability,but it is not optimal in the transient phase of particular plant output.The PID control system can adjust the controller’s gains for optimal hovering(or stable slow flight)and is especially useful for the tracking system.Finally,comparing aircraft stability using PID and LQR controllers shows that the latter has less overshoot and a shorter settling time;in addition,all proposed controllers can be practically deployed as one UAV’s system,which can be handled as an exemplary model of the UAV flight management system.

Microseismic monitoring and forecasting of dynamic disasters in underground hydropower projects in southwest China:A review

The underground hydropower projects in Southwest China is characterized by large excavation sizes,high geostresses,complicated geological conditions and multiple construction processes.Various disasters such as collapses,large deformations,rockbursts are frequently encountered,resulting in serious casualties and huge economic losses.This review mainly presents some representative results on microseismic(MS)monitoring and forecasting for disasters in hydropower underground engineering.First,a set of new denoising,spectral analysis,and location methods were developed for better identification and location of MS signals.Then,the tempo-spatial characteristics of MS events were analyzed to understand the relationship between field construction and damages of surrounding rocks.Combined with field construction,geological data,numerical simulation and parametric analysis of MS sources,the focal mechanism of MS events was revealed.A damage constitutive model considering MS fracturing size was put forward and feedback analysis considering the MS damage of underground surrounding rocks was conducted.Next,an MS multi-parameter based risk assessment and early warning method for dynamic disasters were proposed.The technology for control of the damage and deformation of underground surrounding rocks was proposed for underground caverns.Finally,two typical underground powerhouses were selected as case studies.These achievements can provide significant references for prevention and control of dynamic disasters for underground engineering with similar complicated geological conditions.