Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Mitochondrial dysfunction and quality control lie at the heart of subarachnoid hemorrhage

The dramatic increase in intracranial pressure after subarachnoid hemorrhage leads to a decrease in cerebral perfusion pressure and a reduction in cerebral blood flow.Mitochondria are directly affected by direct factors such as ischemia,hypoxia,excitotoxicity,and toxicity of free hemoglobin and its degradation products,which trigger mitochondrial dysfunction.Dysfunctional mitochondria release large amounts of reactive oxygen species,inflammatory mediators,and apoptotic proteins that activate apoptotic pathways,further damaging cells.In response to this array of damage,cells have adopted multiple mitochondrial quality control mechanisms through evolution,including mitochondrial protein quality control,mitochondrial dynamics,mitophagy,mitochondrial biogenesis,and intercellular mitochondrial transfer,to maintain mitochondrial homeostasis under pathological conditions.Specific interventions targeting mitochondrial quality control mechanisms have emerged as promising therapeutic strategies for subarachnoid hemorrhage.This review provides an overview of recent research advances in mitochondrial pathophysiological processes after subarachnoid hemorrhage,particularly mitochondrial quality control mechanisms.It also presents potential therapeutic strategies to target mitochondrial quality control in subarachnoid hemorrhage.

Communication Resource-Efficient Vehicle Platooning Control With Various Spacing Policies

Platooning represents one of the key features that connected automated vehicles may possess as it allows multiple automated vehicles to be maneuvered cooperatively with small headways on roads. However, a critical challenge in accomplishing automated vehicle platoons is to deal with the effects of intermittent and sporadic vehicle-to-vehicle data transmissions caused by limited wireless communication resources. This paper addresses the co-design problem of dynamic event-triggered communication scheduling and cooperative adaptive cruise control for a convoy of automated vehicles with diverse spacing policies. The central aim is to achieve automated vehicle platooning under various gap references with desired platoon stability and spacing performance requirements, while simultaneously improving communication efficiency. Toward this aim, a dynamic event-triggered scheduling mechanism is developed such that the intervehicle data transmissions are scheduled dynamically and efficiently over time. Then, a tractable co-design criterion on the existence of both the admissible event-driven cooperative adaptive cruise control law and the desired scheduling mechanism is derived. Finally, comparative simulation results are presented to substantiate the effectiveness and merits of the obtained results.

Finite-time Prescribed Performance Time-Varying Formation Control for Second-Order Multi-Agent Systems With Non-Strict Feedback Based on a Neural Network Observer

This paper studies the problem of time-varying formation control with finite-time prescribed performance for nonstrict feedback second-order multi-agent systems with unmeasured states and unknown nonlinearities.To eliminate nonlinearities,neural networks are applied to approximate the inherent dynamics of the system.In addition,due to the limitations of the actual working conditions,each follower agent can only obtain the locally measurable partial state information of the leader agent.To address this problem,a neural network state observer based on the leader state information is designed.Then,a finite-time prescribed performance adaptive output feedback control strategy is proposed by restricting the sliding mode surface to a prescribed region,which ensures that the closed-loop system has practical finite-time stability and that formation errors of the multi-agent systems converge to the prescribed performance bound in finite time.Finally,a numerical simulation is provided to demonstrate the practicality and effectiveness of the developed algorithm.

Vibration Control of A Flexible Marine Riser System Subject to Input Dead Zone and Extraneous Disturbances

An observer-based adaptive backstepping boundary control is proposed for vibration control of flexible offshore riser systems with unknown nonlinear input dead zone and uncertain environmental disturbances.The control algorithm can update the control law online through real-time data to make the controller adapt to the environment and improve the control precision.Specifically,based on the adaptive backstepping framework,virtual control laws and Lyapunov functions are designed for each subsystem.Three direction interference observers are designed to track the timevarying boundary disturbance.On this basis,the inverse of the dead zone and linear state transformation are used to compensate for the original system and eliminate the adverse effects of the dead zone.In addition,the stability of the closed-loop system is proven by Lyapunov stability theory.All the system states are bounded,and the vibration offset of the riser converges to a small area of the initial position.Finally,four examples of flexible marine risers are simulated in MATLAB to verify the effectiveness of the proposed controller.

An Integrated Control Framework for Torque Vectoring and Active Suspension System

Four-wheel independently driven electric vehicles(FWID-EV)endow a flexible and scalable control framework to improve vehicle performance.This paper integrates the torque vectoring and active suspension system(ASS)to enhance the vehicle’s longitudinal and vertical motion control performance.While the nonlinear characteristic of the tire model leads to a relatively heavier computational burden.To facilitate the controller design and ease the load,a half-vehicle dynamics system is built and simplified to the linear-time-varying(LTV)model.Then a model predictive controller is developed by formulating the objective function by comprehensively considering the safety,energy-saving and comfort requirements.The in-wheel motor efficiency and the power loss of tire slip are treated as optimization indices in this work to reduce energy consumption.Finally,the effectiveness of the proposed controller is verified through the rapid-control-prototype(RCP)test.The results demonstrate the enhancement of the energy-saving as well as comfort on the basis of vehicle stability.

A Review on Vibration Control of Multiple Cylinders Subjected to FlowInduced Vibrations

The fatigue damage caused by flow-induced vibration(FIV)is one of the major concerns for multiple cylindrical structures in many engineering applications.The FIV suppression is of great importance for the security of many cylindrical structures.Many active and passive control methods have been employed for the vibration suppression of an isolated cylinder undergoing vortex-induced vibrations(VIV).The FIV suppression methods are mainly extended to the multiple cylinders from the vibration control of the isolated cylinder.Due to the mutual interference between the multiple cylinders,the FIV mechanism is more complex than the VIV mechanism,which makes a great challenge for the FIV suppression.Some efforts have been devoted to vibration suppression of multiple cylinder systems undergoing FIV over the past two decades.The control methods,such as helical strakes,splitter plates,control rods and flexible sheets,are not always effective,depending on many influence factors,such as the spacing ratio,the arrangement geometrical shape,the flow velocity and the parameters of the vibration control devices.The FIV response,hydrodynamic features and wake patterns of the multiple cylinders equipped with vibration control devices are reviewed and summarized.The FIV suppression efficiency of the vibration control methods are analyzed and compared considering different influence factors.Further research on the FIV suppression of multiple cylinders is suggested to provide insight for the development of FIV control methods and promote engineering applications of FIV control methods.

Control of highly pathogenic avian influenza through vaccination

The stamping-out strategy has been used to control highly pathogenic avian influenza viruses in many countries,driven by the belief that vaccination would not be successful against such viruses and fears that avian influenza virus in vaccinated birds would evolve more rapidly and pose a greater risk to humans.In this review,we summarize the successes in controlling highly pathogenic avian influenza in China and make suggestions regarding the requirements for vaccine selection and effectiveness.In addition,we present evidence that vaccination of poultry not only eliminates human infection with avian influenza virus,but also significantly reduces and abolishes some harmful characteristics of avian influenza virus.

A Blockchain-Based Access Control Scheme for Reputation Value Attributes of the Internet of Things

The Internet of Things(IoT)access controlmechanism may encounter security issues such as single point of failure and data tampering.To address these issues,a blockchain-based IoT reputation value attribute access control scheme is proposed.Firstly,writing the reputation value as an attribute into the access control policy,and then deploying the access control policy in the smart contract of the blockchain system can enable the system to provide more fine-grained access control;Secondly,storing a large amount of resources fromthe Internet of Things in Inter Planetary File System(IPFS)to improve system throughput;Finally,map resource access operations to qualification tokens to improve the performance of the access control system.Complete simulation experiments based on the Hyperledger Fabric platform.Fromthe simulation experimental results,it can be seen that the access control system can achieve more fine-grained and dynamic access control while maintaining high throughput and low time delay,providing sufficient reliability and security for access control of IoT devices.

Distributed Platooning Control of Automated Vehicles Subject to Replay Attacks Based on Proportional Integral Observers

Secure platooning control plays an important role in enhancing the cooperative driving safety of automated vehicles subject to various security vulnerabilities.This paper focuses on the distributed secure control issue of automated vehicles affected by replay attacks.A proportional-integral-observer(PIO)with predetermined forgetting parameters is first constructed to acquire the dynamical information of vehicles.Then,a time-varying parameter and two positive scalars are employed to describe the temporal behavior of replay attacks.In light of such a scheme and the common properties of Laplace matrices,the closed-loop system with PIO-based controllers is transformed into a switched and time-delayed one.Furthermore,some sufficient conditions are derived to achieve the desired platooning performance by the view of the Lyapunov stability theory.The controller gains are analytically determined by resorting to the solution of certain matrix inequalities only dependent on maximum and minimum eigenvalues of communication topologies.Finally,a simulation example is provided to illustrate the effectiveness of the proposed control strategy.

A 16-bit 18-MSPS flash-assisted SAR ADC with hybrid synchronous and asynchronous control logic

This paper presents a 16-bit,18-MSPS(million samples per second)flash-assisted successive-approximation-register(SAR)analog-to-digital converter(ADC)utilizing hybrid synchronous and asynchronous(HYSAS)timing control logic based on an on-chip delay-locked loop(DLL).The HYSAS scheme can provide a longer settling time for the capacitive digital-to-analog converter(CDAC)than the synchronous and asynchronous SAR ADC.Therefore,the issue of incomplete settling or ringing in the DAC voltage for cases of either on-chip or off-chip reference voltage can be solved to a large extent.In addition,the fore-ground calibration of the CDAC’s mismatch is performed with a finite-impulse-response bandpass filter(FIR-BPF)based least-mean-square(LMS)algorithm in an off-chip FPGA(field programmable gate array).Fabricated in 40-nm CMOS process,the proto-type ADC achieves 94.02-dB spurious-free dynamic range(SFDR),and 75.98-dB signal-to-noise-and-distortion ratio(SNDR)for a 2.88-MHz input under 18-MSPS sampling rate.

Path Planning and Tracking Control for Parking via Soft Actor-Critic Under Non-Ideal Scenarios

Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.

Preoperative controlling nutritional status as an optimal prognostic nutritional index to predict the outcome for colorectal cancer

BACKGROUND The controlling nutritional status(CONUT)score effectively reflects a patient’s nutritional status,which is closely related to cancer prognosis.This study invest-igated the relationship between the CONUT score and prognosis after radical surgery for colorectal cancer,and compared the predictive ability of the CONUT score with other indexes.AIM To analyze the predictive performance of the CONUT score for the survival rate of colorectal cancer patients who underwent potentially curative resection.METHODS This retrospective analysis included 217 patients with newly diagnosed colorectal.The CONUT score was calculated based on the serum albumin level,total lymphocyte count,and total cholesterol level.The cutoff value of the CONUT score for predicting prognosis was 4 according to the Youden Index by the receiver operating characteristic curve.The associations between the CONUT score and the prognosis were performed using Kaplan-Meier curves and Cox regression analysis.RESULTS Using the cutoff value of the CONUT score,patients were stratified into CONUT low(n=189)and CONUT high groups(n=28).The CONUT high group had worse overall survival(OS)(P=0.013)and relapse-free survival(RFS)(P=0.015).The predictive performance of CONUT was superior to the modified Glasgow prognostic score,the prognostic nutritional index,and the neutrophil-to-lymphocyte ratio.Meanwhile,the predictive performances of CONUT+tumor node metastasis(TNM)stage for 3-year OS[area under the receiver operating characteristics curve(AUC)=0.803]and 3-year RFS(AUC=0.752)were no less than skeletal muscle mass index(SMI)+TNM stage.The CONUT score was negatively correlated with SMI(P<0.01).CONCLUSION As a nutritional indicator,the CONUT score could predict long-term outcomes after radical surgery for colorectal cancer,and its predictive ability was superior to other indexes.The correlation between the CONUT score and skeletal muscle may be one of the factors that play a predictive role.

Adaptive Trajectory Tracking Control for Nonholonomic Wheeled Mobile Robots:A Barrier Function Sliding Mode Approach

The trajectory tracking control performance of nonholonomic wheeled mobile robots(NWMRs)is subject to nonholonomic constraints,system uncertainties,and external disturbances.This paper proposes a barrier function-based adaptive sliding mode control(BFASMC)method to provide high-precision,fast-response performance and robustness for NWMRs.Compared with the conventional adaptive sliding mode control,the proposed control strategy can guarantee that the sliding mode variables converge to a predefined neighborhood of origin with a predefined reaching time independent of the prior knowledge of the uncertainties and disturbances bounds.Another advantage of the proposed algorithm is that the control gains can be adaptively adjusted to follow the disturbances amplitudes thanks to the barrier function.The benefit is that the overestimation of control gain can be eliminated,resulting in chattering reduction.Moreover,a modified barrier function-like control gain is employed to prevent the input saturation problem due to the physical limit of the actuator.The stability analysis and comparative experiments demonstrate that the proposed BFASMC can ensure the prespecified convergence performance of the NWMR system output variables and strong robustness against uncertainties/disturbances.

Hybrid Dynamic Variables-Dependent Event-Triggered Fuzzy Model Predictive Control

This article focuses on dynamic event-triggered mechanism(DETM)-based model predictive control(MPC) for T-S fuzzy systems.A hybrid dynamic variables-dependent DETM is carefully devised,which includes a multiplicative dynamic variable and an additive dynamic variable.The addressed DETM-based fuzzy MPC issue is described as a “min-max” optimization problem(OP).To facilitate the co-design of the MPC controller and the weighting matrix of the DETM,an auxiliary OP is proposed based on a new Lyapunov function and a new robust positive invariant(RPI) set that contain the membership functions and the hybrid dynamic variables.A dynamic event-triggered fuzzy MPC algorithm is developed accordingly,whose recursive feasibility is analysed by employing the RPI set.With the designed controller,the involved fuzzy system is ensured to be asymptotically stable.Two examples show that the new DETM and DETM-based MPC algorithm have the advantages of reducing resource consumption while yielding the anticipated performance.

Analysis of debris flow control effect and hazard assessment in Xinqiao Gully,Wenchuan M_(s)8.0 earthquake area based on numerical simulation

Xinqiao Gully is located in the area of the 2008 Wenchuan M_(s)8.0 earthquake in Sichuan province,China.Based on the investigation of the 2023"6-26"Xinqiao Gully debris flow event,this study assessed the effectiveness of the debris flow control project and evaluated the debris flow hazards.Through field investigation and numerical simulation methods,the indicators of flow intensity reduction rate and storage capacity fullness were proposed to quantify the effectiveness of the engineering measures in the debris flow event.The simulation results show that the debris flow control project reduced the flow intensity by41.05%to 64.61%.The storage capacity of the dam decreases gradually from upstream to the mouth of the gully,thus effectively intercepting and controlling the debris flow.By evaluating the debris flow of different recurrence intervals,further measures are recommended for managing debris flow events.

Application of Specialized Group Management in the Quality Control of Perioperative Nursing

Objective:To explore the role of specialized group management in the quality control of perioperative nursing.Methods:45 surgical nurses from our hospital were selected as the research subjects.Traditional operating room management was adopted from July 2019 to June 2020,and specialized group management was adopted from July 2020 to June 2021.The surgeon’s satisfaction,surgical nurses’core professional competence,and surgical patients’satisfaction were obtained through surveys and the results were analyzed.Results:Surgeon satisfaction before the implementation of specialized group management was significantly lower than after its implementation(P<0.05).Besides,surgical nurses’core professional competency scores before the implementation of specialized group management were significantly lower than after its implementation(P<0.05).Lastly,surgical patients’satisfaction before the implementation of specialized group management was significantly lower than after its implementation(P<0.05).Conclusion:Specialized group management helps to improve the quality of perioperative care and should be applied in clinical practice.

A Tutorial on Quantized Feedback Control

In this tutorial paper, we explore the field of quantized feedback control, which has gained significant attention due to the growing prevalence of networked control systems. These systems require the transmission of feedback information, such as measurements and control signals, over digital networks, presenting novel challenges in estimation and control design. Our examination encompasses various topics, including the minimal information needed for effective feedback control, the design of quantizers, strategies for quantized control design and estimation,achieving consensus control with quantized data, and the pursuit of high-precision tracking using quantized measurements.

Quantization and Event-Triggered Policy Design for Encrypted Networked Control

This paper proposes a novel event-driven encrypted control framework for linear networked control systems(NCSs),which relies on two modified uniform quantization policies,the Paillier cryptosystem,and an event-triggered strategy.Due to the fact that only integers can work in the Pailler cryptosystem,both the real-valued control gain and system state need to be first quantized before encryption.This is dramatically different from the existing quantized control methods,where only the quantization of a single value,e.g.,the control input or the system state,is considered.To handle this issue,static and dynamic quantization policies are presented,which achieve the desired integer conversions and guarantee asymptotic convergence of the quantized system state to the equilibrium.Then,the quantized system state is encrypted and sent to the controller when the triggering condition,specified by a state-based event-triggered strategy,is satisfied.By doing so,not only the security and confidentiality of data transmitted over the communication network are protected,but also the ciphertext expansion phenomenon can be relieved.Additionally,by tactfully designing the quantization sensitivities and triggering error,the proposed event-driven encrypted control framework ensures the asymptotic stability of the overall closedloop system.Finally,a simulation example of the secure motion control for an inverted pendulum cart system is presented to evaluate the effectiveness of the theoretical results.

Noise-Tolerant ZNN-Based Data-Driven Iterative Learning Control for Discrete Nonaffine Nonlinear MIMO Repetitive Systems

Aiming at the tracking problem of a class of discrete nonaffine nonlinear multi-input multi-output(MIMO) repetitive systems subjected to separable and nonseparable disturbances, a novel data-driven iterative learning control(ILC) scheme based on the zeroing neural networks(ZNNs) is proposed. First, the equivalent dynamic linearization data model is obtained by means of dynamic linearization technology, which exists theoretically in the iteration domain. Then, the iterative extended state observer(IESO) is developed to estimate the disturbance and the coupling between systems, and the decoupled dynamic linearization model is obtained for the purpose of controller synthesis. To solve the zero-seeking tracking problem with inherent tolerance of noise,an ILC based on noise-tolerant modified ZNN is proposed. The strict assumptions imposed on the initialization conditions of each iteration in the existing ILC methods can be absolutely removed with our method. In addition, theoretical analysis indicates that the modified ZNN can converge to the exact solution of the zero-seeking tracking problem. Finally, a generalized example and an application-oriented example are presented to verify the effectiveness and superiority of the proposed process.