Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Mitochondrial dysfunction and quality control lie at the heart of subarachnoid hemorrhage

The dramatic increase in intracranial pressure after subarachnoid hemorrhage leads to a decrease in cerebral perfusion pressure and a reduction in cerebral blood flow.Mitochondria are directly affected by direct factors such as ischemia,hypoxia,excitotoxicity,and toxicity of free hemoglobin and its degradation products,which trigger mitochondrial dysfunction.Dysfunctional mitochondria release large amounts of reactive oxygen species,inflammatory mediators,and apoptotic proteins that activate apoptotic pathways,further damaging cells.In response to this array of damage,cells have adopted multiple mitochondrial quality control mechanisms through evolution,including mitochondrial protein quality control,mitochondrial dynamics,mitophagy,mitochondrial biogenesis,and intercellular mitochondrial transfer,to maintain mitochondrial homeostasis under pathological conditions.Specific interventions targeting mitochondrial quality control mechanisms have emerged as promising therapeutic strategies for subarachnoid hemorrhage.This review provides an overview of recent research advances in mitochondrial pathophysiological processes after subarachnoid hemorrhage,particularly mitochondrial quality control mechanisms.It also presents potential therapeutic strategies to target mitochondrial quality control in subarachnoid hemorrhage.

Vibration Control of A Flexible Marine Riser System Subject to Input Dead Zone and Extraneous Disturbances

An observer-based adaptive backstepping boundary control is proposed for vibration control of flexible offshore riser systems with unknown nonlinear input dead zone and uncertain environmental disturbances.The control algorithm can update the control law online through real-time data to make the controller adapt to the environment and improve the control precision.Specifically,based on the adaptive backstepping framework,virtual control laws and Lyapunov functions are designed for each subsystem.Three direction interference observers are designed to track the timevarying boundary disturbance.On this basis,the inverse of the dead zone and linear state transformation are used to compensate for the original system and eliminate the adverse effects of the dead zone.In addition,the stability of the closed-loop system is proven by Lyapunov stability theory.All the system states are bounded,and the vibration offset of the riser converges to a small area of the initial position.Finally,four examples of flexible marine risers are simulated in MATLAB to verify the effectiveness of the proposed controller.

Communication Resource-Efficient Vehicle Platooning Control With Various Spacing Policies

Platooning represents one of the key features that connected automated vehicles may possess as it allows multiple automated vehicles to be maneuvered cooperatively with small headways on roads. However, a critical challenge in accomplishing automated vehicle platoons is to deal with the effects of intermittent and sporadic vehicle-to-vehicle data transmissions caused by limited wireless communication resources. This paper addresses the co-design problem of dynamic event-triggered communication scheduling and cooperative adaptive cruise control for a convoy of automated vehicles with diverse spacing policies. The central aim is to achieve automated vehicle platooning under various gap references with desired platoon stability and spacing performance requirements, while simultaneously improving communication efficiency. Toward this aim, a dynamic event-triggered scheduling mechanism is developed such that the intervehicle data transmissions are scheduled dynamically and efficiently over time. Then, a tractable co-design criterion on the existence of both the admissible event-driven cooperative adaptive cruise control law and the desired scheduling mechanism is derived. Finally, comparative simulation results are presented to substantiate the effectiveness and merits of the obtained results.

A Review on Vibration Control of Multiple Cylinders Subjected to FlowInduced Vibrations

The fatigue damage caused by flow-induced vibration(FIV)is one of the major concerns for multiple cylindrical structures in many engineering applications.The FIV suppression is of great importance for the security of many cylindrical structures.Many active and passive control methods have been employed for the vibration suppression of an isolated cylinder undergoing vortex-induced vibrations(VIV).The FIV suppression methods are mainly extended to the multiple cylinders from the vibration control of the isolated cylinder.Due to the mutual interference between the multiple cylinders,the FIV mechanism is more complex than the VIV mechanism,which makes a great challenge for the FIV suppression.Some efforts have been devoted to vibration suppression of multiple cylinder systems undergoing FIV over the past two decades.The control methods,such as helical strakes,splitter plates,control rods and flexible sheets,are not always effective,depending on many influence factors,such as the spacing ratio,the arrangement geometrical shape,the flow velocity and the parameters of the vibration control devices.The FIV response,hydrodynamic features and wake patterns of the multiple cylinders equipped with vibration control devices are reviewed and summarized.The FIV suppression efficiency of the vibration control methods are analyzed and compared considering different influence factors.Further research on the FIV suppression of multiple cylinders is suggested to provide insight for the development of FIV control methods and promote engineering applications of FIV control methods.

Evolutionary Decision-Making and Planning for Autonomous Driving Based on Safe and Rational Exploration and Exploitation

Decision-making and motion planning are extremely important in autonomous driving to ensure safe driving in a real-world environment.This study proposes an online evolutionary decision-making and motion planning framework for autonomous driving based on a hybrid data-and model-driven method.First,a data-driven decision-making module based on deep reinforcement learning(DRL)is developed to pursue a rational driving performance as much as possible.Then,model predictive control(MPC)is employed to execute both longitudinal and lateral motion planning tasks.Multiple constraints are defined according to the vehicle’s physical limit to meet the driving task requirements.Finally,two principles of safety and rationality for the self-evolution of autonomous driving are proposed.A motion envelope is established and embedded into a rational exploration and exploitation scheme,which filters out unreasonable experiences by masking unsafe actions so as to collect high-quality training data for the DRL agent.Experiments with a high-fidelity vehicle model and MATLAB/Simulink co-simulation environment are conducted,and the results show that the proposed online-evolution framework is able to generate safer,more rational,and more efficient driving action in a real-world environment.

A 16-bit 18-MSPS flash-assisted SAR ADC with hybrid synchronous and asynchronous control logic

This paper presents a 16-bit,18-MSPS(million samples per second)flash-assisted successive-approximation-register(SAR)analog-to-digital converter(ADC)utilizing hybrid synchronous and asynchronous(HYSAS)timing control logic based on an on-chip delay-locked loop(DLL).The HYSAS scheme can provide a longer settling time for the capacitive digital-to-analog converter(CDAC)than the synchronous and asynchronous SAR ADC.Therefore,the issue of incomplete settling or ringing in the DAC voltage for cases of either on-chip or off-chip reference voltage can be solved to a large extent.In addition,the fore-ground calibration of the CDAC’s mismatch is performed with a finite-impulse-response bandpass filter(FIR-BPF)based least-mean-square(LMS)algorithm in an off-chip FPGA(field programmable gate array).Fabricated in 40-nm CMOS process,the proto-type ADC achieves 94.02-dB spurious-free dynamic range(SFDR),and 75.98-dB signal-to-noise-and-distortion ratio(SNDR)for a 2.88-MHz input under 18-MSPS sampling rate.

Path Planning and Tracking Control for Parking via Soft Actor-Critic Under Non-Ideal Scenarios

Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.

The pathogenic mechanism of TAR DNA-binding protein 43(TDP-43)in amyotrophic lateral sclerosis

The onset of amyotrophic lateral sclerosis is usually characterized by focal death of both upper and/or lower motor neurons occurring in the motor cortex,basal ganglia,brainstem,and spinal cord,and commonly involves the muscles of the upper and/or lower extremities,and the muscles of the bulbar and/or respiratory regions.However,as the disease progresses,it affects the adjacent body regions,leading to generalized muscle weakness,occasionally along with memory,cognitive,behavioral,and language impairments;respiratory dysfunction occurs at the final stage of the disease.The disease has a complicated pathophysiology and currently,only riluzole,edaravone,and phenylbutyrate/taurursodiol are licensed to treat amyotrophic lateral sclerosis in many industrialized countries.The TAR DNA-binding protein 43 inclusions are observed in 97%of those diagnosed with amyotrophic lateral sclerosis.This review provides a preliminary overview of the potential effects of TAR DNAbinding protein 43 in the pathogenesis of amyotrophic lateral sclerosis,including the abnormalities in nucleoplasmic transport,RNA function,post-translational modification,liquid-liquid phase separation,stress granules,mitochondrial dysfunction,oxidative stress,axonal transport,protein quality control system,and non-cellular autonomous functions(e.g.,glial cell functions and prion-like propagation).

混合磁轴承PID控制参数整定方法研究

针对混合磁轴承PID控制器参数整定复杂的问题,分别采用PID Tuner和Controller System Tuner工具箱两种整定方法对PID参数进行整定,观察混合磁轴承系统的控制效果,并给定恒定外部干扰力对得到的PID参数进行验证,通过仿真结果证明了两种参数整定方法的有效性和实用性。

A Tutorial on Quantized Feedback Control

In this tutorial paper, we explore the field of quantized feedback control, which has gained significant attention due to the growing prevalence of networked control systems. These systems require the transmission of feedback information, such as measurements and control signals, over digital networks, presenting novel challenges in estimation and control design. Our examination encompasses various topics, including the minimal information needed for effective feedback control, the design of quantizers, strategies for quantized control design and estimation,achieving consensus control with quantized data, and the pursuit of high-precision tracking using quantized measurements.

Finite-time Prescribed Performance Time-Varying Formation Control for Second-Order Multi-Agent Systems With Non-Strict Feedback Based on a Neural Network Observer

This paper studies the problem of time-varying formation control with finite-time prescribed performance for nonstrict feedback second-order multi-agent systems with unmeasured states and unknown nonlinearities.To eliminate nonlinearities,neural networks are applied to approximate the inherent dynamics of the system.In addition,due to the limitations of the actual working conditions,each follower agent can only obtain the locally measurable partial state information of the leader agent.To address this problem,a neural network state observer based on the leader state information is designed.Then,a finite-time prescribed performance adaptive output feedback control strategy is proposed by restricting the sliding mode surface to a prescribed region,which ensures that the closed-loop system has practical finite-time stability and that formation errors of the multi-agent systems converge to the prescribed performance bound in finite time.Finally,a numerical simulation is provided to demonstrate the practicality and effectiveness of the developed algorithm.

Quantization and Event-Triggered Policy Design for Encrypted Networked Control

This paper proposes a novel event-driven encrypted control framework for linear networked control systems(NCSs),which relies on two modified uniform quantization policies,the Paillier cryptosystem,and an event-triggered strategy.Due to the fact that only integers can work in the Pailler cryptosystem,both the real-valued control gain and system state need to be first quantized before encryption.This is dramatically different from the existing quantized control methods,where only the quantization of a single value,e.g.,the control input or the system state,is considered.To handle this issue,static and dynamic quantization policies are presented,which achieve the desired integer conversions and guarantee asymptotic convergence of the quantized system state to the equilibrium.Then,the quantized system state is encrypted and sent to the controller when the triggering condition,specified by a state-based event-triggered strategy,is satisfied.By doing so,not only the security and confidentiality of data transmitted over the communication network are protected,but also the ciphertext expansion phenomenon can be relieved.Additionally,by tactfully designing the quantization sensitivities and triggering error,the proposed event-driven encrypted control framework ensures the asymptotic stability of the overall closedloop system.Finally,a simulation example of the secure motion control for an inverted pendulum cart system is presented to evaluate the effectiveness of the theoretical results.

Ensuring Secure Platooning of Constrained Intelligent and Connected Vehicles Against Byzantine Attacks:A Distributed MPC Framework

This study investigates resilient platoon control for constrained intelligent and connected vehicles(ICVs)against F-local Byzantine attacks.We introduce a resilient distributed model-predictive platooning control framework for such ICVs.This framework seamlessly integrates the predesigned optimal control with distributed model predictive control(DMPC)optimization and introduces a unique distributed attack detector to ensure the reliability of the transmitted information among vehicles.Notably,our strategy uses previously broadcasted information and a specialized convex set,termed the“resilience set”,to identify unreliable data.This approach significantly eases graph robustness prerequisites,requiring only an(F+1)-robust graph,in contrast to the established mean sequence reduced algorithms,which require a minimum(2F+1)-robust graph.Additionally,we introduce a verification algorithm to restore trust in vehicles under minor attacks,further reducing communication network robustness.Our analysis demonstrates the recursive feasibility of the DMPC optimization.Furthermore,the proposed method achieves exceptional control performance by minimizing the discrepancies between the DMPC control inputs and predesigned platoon control inputs,while ensuring constraint compliance and cybersecurity.Simulation results verify the effectiveness of our theoretical findings.

Noise-Tolerant ZNN-Based Data-Driven Iterative Learning Control for Discrete Nonaffine Nonlinear MIMO Repetitive Systems

Aiming at the tracking problem of a class of discrete nonaffine nonlinear multi-input multi-output(MIMO) repetitive systems subjected to separable and nonseparable disturbances, a novel data-driven iterative learning control(ILC) scheme based on the zeroing neural networks(ZNNs) is proposed. First, the equivalent dynamic linearization data model is obtained by means of dynamic linearization technology, which exists theoretically in the iteration domain. Then, the iterative extended state observer(IESO) is developed to estimate the disturbance and the coupling between systems, and the decoupled dynamic linearization model is obtained for the purpose of controller synthesis. To solve the zero-seeking tracking problem with inherent tolerance of noise,an ILC based on noise-tolerant modified ZNN is proposed. The strict assumptions imposed on the initialization conditions of each iteration in the existing ILC methods can be absolutely removed with our method. In addition, theoretical analysis indicates that the modified ZNN can converge to the exact solution of the zero-seeking tracking problem. Finally, a generalized example and an application-oriented example are presented to verify the effectiveness and superiority of the proposed process.

Command filtered integrated estimation guidance and control for strapdown missiles with circular field of view

In this paper,an integrated estimation guidance and control(IEGC)system is designed based on the command filtered backstepping approach for circular field-of-view(FOV)strapdown missiles.The threedimensional integrated estimation guidance and control nonlinear model with limited actuator deflection angle is established considering the seeker's FOV constraint.The boundary time-varying integral barrier Lyapunov function(IBLF)is employed in backstepping design to constrain the body line-of-sight(BLOS)in IEGC system to fit a circular FOV.Then,the nonlinear adaptive controller is designed to estimate the changing aerodynamic parameters.The generalized extended state observer(GESO)is designed to estimate the acceleration of the maneuvering targets and the unmatched time-varying disturbances for improving tracking accuracy.Furthermore,the command filters are used to solve the"differential expansion"problem during the backstepping design.The Lyapunov theory is used to prove the stability of the overall closed-loop IEGC system.Finally,the simulation results validate the integrated system's effectiveness,achieving high accuracy strikes against maneuvering targets.

Practical Prescribed Time Tracking Control With Bounded Time-Varying Gain Under Non-Vanishing Uncertainties

This paper investigates the prescribed-time control(PTC) problem for a class of strict-feedback systems subject to non-vanishing uncertainties. The coexistence of mismatched uncertainties and non-vanishing disturbances makes PTC synthesis nontrivial. In this work, a control method that does not involve infinite time-varying gain is proposed, leading to a practical and global prescribed time tracking control solution for the strict-feedback systems, in spite of both the mismatched and nonvanishing uncertainties. Different from methods based on control switching to avoid the issue of infinite control gain that involves control discontinuity at the switching point, in our method a softening unit is exclusively included to ensure the continuity of the control action. Furthermore, in contrast to most existing prescribed-time control works where the control scheme is only valid on a finite time interval, in this work, the proposed control scheme is valid on the entire time interval. In addition, the prior information on the upper or lower bound of gi is not in need,enlarging the applicability of the proposed method. Both the theoretical analysis and numerical simulation confirm the effectiveness of the proposed control algorithm.

Nonlinear robust adaptive control for bidirectional stabilization system of all-electric tank with unknown actuator backlash compensation and disturbance estimation

Since backlash nonlinearity is inevitably existing in actuators for bidirectional stabilization system of allelectric tank,it behaves more drastically in high maneuvering environments.In this work,the accurate tracking control for bidirectional stabilization system of moving all-electric tank with actuator backlash and unmodeled disturbance is solved.By utilizing the smooth adaptive backlash inverse model,a nonlinear robust adaptive feedback control scheme is presented.The unknown parameters and unmodelled disturbance are addressed separately through the derived parametric adaptive function and the continuous nonlinear robust term.Because the unknown backlash parameters are updated via adaptive function and the backlash effect can be suppressed successfully by inverse operation,which ensures the system stability.Meanwhile,the system disturbance in the high maneuverable environment can be estimated with the constructed adaptive law online improving the engineering practicality.Finally,Lyapunov-based analysis proves that the developed controller can ensure the tracking error asymptotically converges to zero even with unmodeled disturbance and unknown actuator backlash.Contrast co-simulations and experiments illustrate the advantages of the proposed approach.

Quaternion-Based Adaptive Trajectory Tracking Control of a Rotor-Missile with Unknown Parameters Identification

This paper investigates the adaptive trajectory tracking control problem and the unknown parameter identification problem of a class of rotor-missiles with parametric system uncertainties.First,considering the uncertainty of structural and aerodynamic parameters,the six-degree-of-freedom(6Do F) nonlinear equations describing the position and attitude dynamics of the rotor-missile are established,respectively,in the inertial and body-fixed reference frames.Next,a hierarchical adaptive trajectory tracking controller that can guarantee closed-loop stability is proposed according to the cascade characteristics of the 6Do F dynamics.Then,a memory-augmented update rule of unknown parameters is proposed by integrating all historical data of the regression matrix.As long as the finitely excited condition is satisfied,the precise identification of unknown parameters can be achieved.Finally,the validity of the proposed trajectory tracking controller and the parameter identification method is proved through Lyapunov stability theory and numerical simulations.

Versatile and controlled quantum teleportation network

A quantum teleportation network involving multiple users is essential for future quantum internet.So far,controlled quantum teleportation has been demonstrated in a three-user network.However,versatile and controlled quantum teleportation network involving more users is in demand,which satisfies different combinations of users for practical requirements.Here we propose a highly versatile and controlled teleportation network that can switch among various combinations of different users.We use a single continuous-variable six-partite Greenberger-Horne-Zeilinger(GHZ)state to realize such a task by choosing the different measurement and feedback operations.The controlled teleportation network,which includes one sub-network,two sub-networks and three sub-networks,can be realized for different application of user combinations.Furthermore,the coherent feedback control(CFC)can manipulate and improve the teleportation performance.Our approach is flexible and scalable,and would provide a versatile platform for demonstrations of complex quantum communication and quantum computing protocols.