Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Patient satisfaction and follow-up adherence to glaucoma case management clinic in China

AIM:To assess glaucoma patient satisfaction and follow-up adherence in case management and identify associated predictors to improve healthcare quality and patient outcomes.METHODS:In this cross-sectional study,a total of 119 patients completed a Patient Satisfaction Questionnaire-18 and a sociodemographic questionnaire.Clinical data was obtained from the case management system.Follow-up adherence was defined as completing each follow-up within±30d of the scheduled time set by ophthalmologists during the study period.RESULTS:Average satisfaction scored 78.65±7,with an average of 4.39±0.58 across the seven dimensions.Age negatively correlated with satisfaction(P=0.008),whilst patients with follow-up duration of 2 or more years reported higher satisfaction(P=0.045).Multivariate logistics regression analysis revealed that longer follow-up durations were associated with lower follow-up adherence(OR=0.97,95%CI,0.95-1.00,P=0.044).Additionally,patients with suspected glaucoma(OR=2.72,95%CI,1.03-7.20,P=0.044)and those with an annual income over 100000 Chinese yuan demonstrated higher adherence(OR=5.57,95%CI,1.00-30.89,P=0.049).CONCLUSION:The case management model proves effective for glaucoma patients,with positive adherence rates.The implementation of this model can be optimized in the future based on the identified factors and extended to glaucoma patients in more hospitals.

Vibration Control of A Flexible Marine Riser System Subject to Input Dead Zone and Extraneous Disturbances

An observer-based adaptive backstepping boundary control is proposed for vibration control of flexible offshore riser systems with unknown nonlinear input dead zone and uncertain environmental disturbances.The control algorithm can update the control law online through real-time data to make the controller adapt to the environment and improve the control precision.Specifically,based on the adaptive backstepping framework,virtual control laws and Lyapunov functions are designed for each subsystem.Three direction interference observers are designed to track the timevarying boundary disturbance.On this basis,the inverse of the dead zone and linear state transformation are used to compensate for the original system and eliminate the adverse effects of the dead zone.In addition,the stability of the closed-loop system is proven by Lyapunov stability theory.All the system states are bounded,and the vibration offset of the riser converges to a small area of the initial position.Finally,four examples of flexible marine risers are simulated in MATLAB to verify the effectiveness of the proposed controller.

Control of highly pathogenic avian influenza through vaccination

The stamping-out strategy has been used to control highly pathogenic avian influenza viruses in many countries,driven by the belief that vaccination would not be successful against such viruses and fears that avian influenza virus in vaccinated birds would evolve more rapidly and pose a greater risk to humans.In this review,we summarize the successes in controlling highly pathogenic avian influenza in China and make suggestions regarding the requirements for vaccine selection and effectiveness.In addition,we present evidence that vaccination of poultry not only eliminates human infection with avian influenza virus,but also significantly reduces and abolishes some harmful characteristics of avian influenza virus.

A Blockchain-Based Access Control Scheme for Reputation Value Attributes of the Internet of Things

The Internet of Things(IoT)access controlmechanism may encounter security issues such as single point of failure and data tampering.To address these issues,a blockchain-based IoT reputation value attribute access control scheme is proposed.Firstly,writing the reputation value as an attribute into the access control policy,and then deploying the access control policy in the smart contract of the blockchain system can enable the system to provide more fine-grained access control;Secondly,storing a large amount of resources fromthe Internet of Things in Inter Planetary File System(IPFS)to improve system throughput;Finally,map resource access operations to qualification tokens to improve the performance of the access control system.Complete simulation experiments based on the Hyperledger Fabric platform.Fromthe simulation experimental results,it can be seen that the access control system can achieve more fine-grained and dynamic access control while maintaining high throughput and low time delay,providing sufficient reliability and security for access control of IoT devices.

Mitochondrial dysfunction and quality control lie at the heart of subarachnoid hemorrhage

The dramatic increase in intracranial pressure after subarachnoid hemorrhage leads to a decrease in cerebral perfusion pressure and a reduction in cerebral blood flow.Mitochondria are directly affected by direct factors such as ischemia,hypoxia,excitotoxicity,and toxicity of free hemoglobin and its degradation products,which trigger mitochondrial dysfunction.Dysfunctional mitochondria release large amounts of reactive oxygen species,inflammatory mediators,and apoptotic proteins that activate apoptotic pathways,further damaging cells.In response to this array of damage,cells have adopted multiple mitochondrial quality control mechanisms through evolution,including mitochondrial protein quality control,mitochondrial dynamics,mitophagy,mitochondrial biogenesis,and intercellular mitochondrial transfer,to maintain mitochondrial homeostasis under pathological conditions.Specific interventions targeting mitochondrial quality control mechanisms have emerged as promising therapeutic strategies for subarachnoid hemorrhage.This review provides an overview of recent research advances in mitochondrial pathophysiological processes after subarachnoid hemorrhage,particularly mitochondrial quality control mechanisms.It also presents potential therapeutic strategies to target mitochondrial quality control in subarachnoid hemorrhage.

A Review on Vibration Control of Multiple Cylinders Subjected to FlowInduced Vibrations

The fatigue damage caused by flow-induced vibration(FIV)is one of the major concerns for multiple cylindrical structures in many engineering applications.The FIV suppression is of great importance for the security of many cylindrical structures.Many active and passive control methods have been employed for the vibration suppression of an isolated cylinder undergoing vortex-induced vibrations(VIV).The FIV suppression methods are mainly extended to the multiple cylinders from the vibration control of the isolated cylinder.Due to the mutual interference between the multiple cylinders,the FIV mechanism is more complex than the VIV mechanism,which makes a great challenge for the FIV suppression.Some efforts have been devoted to vibration suppression of multiple cylinder systems undergoing FIV over the past two decades.The control methods,such as helical strakes,splitter plates,control rods and flexible sheets,are not always effective,depending on many influence factors,such as the spacing ratio,the arrangement geometrical shape,the flow velocity and the parameters of the vibration control devices.The FIV response,hydrodynamic features and wake patterns of the multiple cylinders equipped with vibration control devices are reviewed and summarized.The FIV suppression efficiency of the vibration control methods are analyzed and compared considering different influence factors.Further research on the FIV suppression of multiple cylinders is suggested to provide insight for the development of FIV control methods and promote engineering applications of FIV control methods.

A 16-bit 18-MSPS flash-assisted SAR ADC with hybrid synchronous and asynchronous control logic

This paper presents a 16-bit,18-MSPS(million samples per second)flash-assisted successive-approximation-register(SAR)analog-to-digital converter(ADC)utilizing hybrid synchronous and asynchronous(HYSAS)timing control logic based on an on-chip delay-locked loop(DLL).The HYSAS scheme can provide a longer settling time for the capacitive digital-to-analog converter(CDAC)than the synchronous and asynchronous SAR ADC.Therefore,the issue of incomplete settling or ringing in the DAC voltage for cases of either on-chip or off-chip reference voltage can be solved to a large extent.In addition,the fore-ground calibration of the CDAC’s mismatch is performed with a finite-impulse-response bandpass filter(FIR-BPF)based least-mean-square(LMS)algorithm in an off-chip FPGA(field programmable gate array).Fabricated in 40-nm CMOS process,the proto-type ADC achieves 94.02-dB spurious-free dynamic range(SFDR),and 75.98-dB signal-to-noise-and-distortion ratio(SNDR)for a 2.88-MHz input under 18-MSPS sampling rate.

Communication Resource-Efficient Vehicle Platooning Control With Various Spacing Policies

Platooning represents one of the key features that connected automated vehicles may possess as it allows multiple automated vehicles to be maneuvered cooperatively with small headways on roads. However, a critical challenge in accomplishing automated vehicle platoons is to deal with the effects of intermittent and sporadic vehicle-to-vehicle data transmissions caused by limited wireless communication resources. This paper addresses the co-design problem of dynamic event-triggered communication scheduling and cooperative adaptive cruise control for a convoy of automated vehicles with diverse spacing policies. The central aim is to achieve automated vehicle platooning under various gap references with desired platoon stability and spacing performance requirements, while simultaneously improving communication efficiency. Toward this aim, a dynamic event-triggered scheduling mechanism is developed such that the intervehicle data transmissions are scheduled dynamically and efficiently over time. Then, a tractable co-design criterion on the existence of both the admissible event-driven cooperative adaptive cruise control law and the desired scheduling mechanism is derived. Finally, comparative simulation results are presented to substantiate the effectiveness and merits of the obtained results.

Path Planning and Tracking Control for Parking via Soft Actor-Critic Under Non-Ideal Scenarios

Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.

A Tutorial on Quantized Feedback Control

In this tutorial paper, we explore the field of quantized feedback control, which has gained significant attention due to the growing prevalence of networked control systems. These systems require the transmission of feedback information, such as measurements and control signals, over digital networks, presenting novel challenges in estimation and control design. Our examination encompasses various topics, including the minimal information needed for effective feedback control, the design of quantizers, strategies for quantized control design and estimation,achieving consensus control with quantized data, and the pursuit of high-precision tracking using quantized measurements.

Research on the Follow-Up Control Strategy of Biaxial Fatigue Test of Wind Turbine Blade Based on Electromagnetic Excitation

Aiming at the drift problem that the tracking control of the actual load relative to the target load during the electromagnetic excitation biaxial fatigue test of wind turbine blades is easy to drift,a biaxial fatigue testingmachine for electromagnetic excitation is designed,and the following strategy of the actual load and the target load is studied.A Fast Transversal Recursive Least Squares algorithm based on fuzzy logic(Fuzzy FTRLS)is proposed to develop a fatigue loading following dynamic strategy,which adjusts the forgetting factor in the algorithmthrough fuzzy logic to overcome the contradiction between convergence accuracy and convergence speed and solve the phenomenon of amplitude overshoot and phase lag of the actual load relative to the target load.Combined with the previous research results,a simulation model was constructed to verify the strategy’s effectiveness.Field tests were carried out to verify its follow-up effect.The results showthat the tracking error of flapwise and edgewise direction iswithin 4%,which has better robustness and dynamic and static performance than the traditional Recursive Least Squares(RLS)algorithm.

Loss to Specialized Cardiology Follow-Up in Adults Living with Congenital Heart Disease

Background:Much has been written about the loss to follow-up in the transition between pediatric and adult Congenital Heart Disease(CHD)care centers.Much less is understood about the loss to follow-up(LTF)after a successful transition.This is critical too,as patients lost to specialised care are more likely to experience mor-bidity and premature mortality.Aims:To understand the prevalence and reasons for loss to follow-up(LTF)at a large Australian Adult Congenital Heart Disease(ACHD)centre.Methods:Patients with moderate or highly complex CHD and gaps in care of>3 years(defined as LTF)were identified from a comprehensive ACHD data-base.Structured telephone interviews examined current care and barriers to clinic attendance.Results:Overall,407(22%)of ACHD patients(n=1842)were LTF.The mean age at LTF was 31(SD 11.5)years and 54%were male;311(76%)were uncontactable.Compared to adults seen regularly,lost patients were younger,with a greater socio-economic disadvantage,and had less complex CHD(p<0.05 for all).We interviewed 59 patients(14%).The top 3 responses for care absences were“feeling well”(61%),losing track of time(36%),and not needing fol-low-up care(25%).Conclusions:A large proportion of the ACHD population becomes lost to specialised cardiac care,even after a successful transition.This Australian study reports younger age,moderate complexity defects,and socio-economic disadvantage as predictive of loss to follow-up.This study highlights the need for novel approaches to patient-centered service delivery even beyond the age of transition and resources to maintain patient engagement within the ACHD service.

Risk-Informed Model-Free Safe Control of Linear Parameter-Varying Systems

This paper presents a risk-informed data-driven safe control design approach for a class of stochastic uncertain nonlinear discrete-time systems.The nonlinear system is modeled using linear parameter-varying(LPV)systems.A model-based probabilistic safe controller is first designed to guarantee probabilisticλ-contractivity(i.e.,stability and invariance)of the LPV system with respect to a given polyhedral safe set.To obviate the requirement of knowing the LPV system model and to bypass identifying its open-loop model,its closed-loop data-based representation is provided in terms of state and scheduling data as well as a decision variable.It is shown that the variance of the closedloop system,as well as the probability of safety satisfaction,depends on the decision variable and the noise covariance.A minimum-variance direct data-driven gain-scheduling safe control design approach is presented next by designing the decision variable such that all possible closed-loop system realizations satisfy safety with the highest confidence level.This minimum-variance approach is a control-oriented learning method since it minimizes the variance of the state of the closed-loop system with respect to the safe set,and thus minimizes the risk of safety violation.Unlike the certainty-equivalent approach that results in a risk-neutral control design,the minimum-variance method leads to a risk-averse control design.It is shown that the presented direct risk-averse learning approach requires weaker data richness conditions than existing indirect learning methods based on system identification and can lead to a lower risk of safety violation.Two simulation examples along with an experimental validation on an autonomous vehicle are provided to show the effectiveness of the presented approach.

Essential role of postoperative follow-up in the management of clear cell sarcoma

Clear cell sarcoma(CCS)is a rare melanocytic soft tissue sarcoma known for itspropensity to metastasize to the lymph nodes and typically has an unfavorableprognosis.Currently,surgical resection is the primary treatment for localizedCCS,while radiotherapy and chemotherapy are preferred for metastatic cases.The roles of adjuvant chemotherapy,radiotherapy,and lymph node dissection arecontroversial.Although immunotherapy has emerged as a promising avenue inCCS treatment research,there are no established clinical standards for postoperativefollow-up.This editorial discusses a recent article by Liu et al,with afocus on current diagnostic modalities,treatment approaches,and the challengingprognosis associated with CCS.Our aim is to underscore the importance of longtermpatient follow-up in CCS management.

Finite-time Prescribed Performance Time-Varying Formation Control for Second-Order Multi-Agent Systems With Non-Strict Feedback Based on a Neural Network Observer

This paper studies the problem of time-varying formation control with finite-time prescribed performance for nonstrict feedback second-order multi-agent systems with unmeasured states and unknown nonlinearities.To eliminate nonlinearities,neural networks are applied to approximate the inherent dynamics of the system.In addition,due to the limitations of the actual working conditions,each follower agent can only obtain the locally measurable partial state information of the leader agent.To address this problem,a neural network state observer based on the leader state information is designed.Then,a finite-time prescribed performance adaptive output feedback control strategy is proposed by restricting the sliding mode surface to a prescribed region,which ensures that the closed-loop system has practical finite-time stability and that formation errors of the multi-agent systems converge to the prescribed performance bound in finite time.Finally,a numerical simulation is provided to demonstrate the practicality and effectiveness of the developed algorithm.

Quantization and Event-Triggered Policy Design for Encrypted Networked Control

This paper proposes a novel event-driven encrypted control framework for linear networked control systems(NCSs),which relies on two modified uniform quantization policies,the Paillier cryptosystem,and an event-triggered strategy.Due to the fact that only integers can work in the Pailler cryptosystem,both the real-valued control gain and system state need to be first quantized before encryption.This is dramatically different from the existing quantized control methods,where only the quantization of a single value,e.g.,the control input or the system state,is considered.To handle this issue,static and dynamic quantization policies are presented,which achieve the desired integer conversions and guarantee asymptotic convergence of the quantized system state to the equilibrium.Then,the quantized system state is encrypted and sent to the controller when the triggering condition,specified by a state-based event-triggered strategy,is satisfied.By doing so,not only the security and confidentiality of data transmitted over the communication network are protected,but also the ciphertext expansion phenomenon can be relieved.Additionally,by tactfully designing the quantization sensitivities and triggering error,the proposed event-driven encrypted control framework ensures the asymptotic stability of the overall closedloop system.Finally,a simulation example of the secure motion control for an inverted pendulum cart system is presented to evaluate the effectiveness of the theoretical results.

Practical Prescribed Time Tracking Control With Bounded Time-Varying Gain Under Non-Vanishing Uncertainties

This paper investigates the prescribed-time control(PTC) problem for a class of strict-feedback systems subject to non-vanishing uncertainties. The coexistence of mismatched uncertainties and non-vanishing disturbances makes PTC synthesis nontrivial. In this work, a control method that does not involve infinite time-varying gain is proposed, leading to a practical and global prescribed time tracking control solution for the strict-feedback systems, in spite of both the mismatched and nonvanishing uncertainties. Different from methods based on control switching to avoid the issue of infinite control gain that involves control discontinuity at the switching point, in our method a softening unit is exclusively included to ensure the continuity of the control action. Furthermore, in contrast to most existing prescribed-time control works where the control scheme is only valid on a finite time interval, in this work, the proposed control scheme is valid on the entire time interval. In addition, the prior information on the upper or lower bound of gi is not in need,enlarging the applicability of the proposed method. Both the theoretical analysis and numerical simulation confirm the effectiveness of the proposed control algorithm.

Noise-Tolerant ZNN-Based Data-Driven Iterative Learning Control for Discrete Nonaffine Nonlinear MIMO Repetitive Systems

Aiming at the tracking problem of a class of discrete nonaffine nonlinear multi-input multi-output(MIMO) repetitive systems subjected to separable and nonseparable disturbances, a novel data-driven iterative learning control(ILC) scheme based on the zeroing neural networks(ZNNs) is proposed. First, the equivalent dynamic linearization data model is obtained by means of dynamic linearization technology, which exists theoretically in the iteration domain. Then, the iterative extended state observer(IESO) is developed to estimate the disturbance and the coupling between systems, and the decoupled dynamic linearization model is obtained for the purpose of controller synthesis. To solve the zero-seeking tracking problem with inherent tolerance of noise,an ILC based on noise-tolerant modified ZNN is proposed. The strict assumptions imposed on the initialization conditions of each iteration in the existing ILC methods can be absolutely removed with our method. In addition, theoretical analysis indicates that the modified ZNN can converge to the exact solution of the zero-seeking tracking problem. Finally, a generalized example and an application-oriented example are presented to verify the effectiveness and superiority of the proposed process.

Quaternion-Based Adaptive Trajectory Tracking Control of a Rotor-Missile with Unknown Parameters Identification

This paper investigates the adaptive trajectory tracking control problem and the unknown parameter identification problem of a class of rotor-missiles with parametric system uncertainties.First,considering the uncertainty of structural and aerodynamic parameters,the six-degree-of-freedom(6Do F) nonlinear equations describing the position and attitude dynamics of the rotor-missile are established,respectively,in the inertial and body-fixed reference frames.Next,a hierarchical adaptive trajectory tracking controller that can guarantee closed-loop stability is proposed according to the cascade characteristics of the 6Do F dynamics.Then,a memory-augmented update rule of unknown parameters is proposed by integrating all historical data of the regression matrix.As long as the finitely excited condition is satisfied,the precise identification of unknown parameters can be achieved.Finally,the validity of the proposed trajectory tracking controller and the parameter identification method is proved through Lyapunov stability theory and numerical simulations.