Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Distributed Robust UAVs Formation Control Based on Semidefinite Programming

The formation control of unmanned aerial vehicle(UAV)swarms is of significant importance in various fields such as transportation,emergency management,and environmental monitoring.However,the complex dynamics,nonlinearity,uncertainty,and interaction among agents make it a challenging problem.In this paper,we propose a distributed robust control strategy that uses only local information of UAVs to improve the stability and robustness of the formation system in uncertain environments.We establish a nominal control strategy based on position relations and a semi-definite programming model to obtain control gains.Additionally,we propose a robust control strategy under the rotation setΩto address the noise and disturbance in the system,ensuring that even when the rotation angles of the UAVs change,they still form a stable formation.Finally,we extend the proposed strategy to a quadrotor UAV system with high-order kinematic models and conduct simulation experiments to validate its effectiveness in resisting uncertain disturbances and achieving formation control.

Complete-basis-reprogrammable coding metasurface for generating dynamicallycontrolled holograms under arbitrary polarization states

Reprogrammable metasurfaces,which establish a fascinating bridge between physical and information domains,can dynamically control electromagnetic(EM)waves in real time and thus have attracted great attentions from researchers around the world.To control EM waves with an arbitrary polarization state,it is desirable that a complete set of basis states be controlled independently since incident EM waves with an arbitrary polarization state can be decomposed as a linear sum of these basis states.In this work,we present the concept of complete-basis-reprogrammable coding metasurface(CBR-CM)in reflective manners,which can achieve independently dynamic controls over the reflection phases while maintaining the same amplitude for left-handed circularly polarized(LCP)waves and right-handed circularly polarized(RCP)waves.Since LCP and RCP waves together constitute a complete basis set of planar EM waves,dynamicallycontrolled holograms can be generated under arbitrarily polarized wave incidence.The dynamically reconfigurable metaparticle is implemented to demonstrate the CBR-CM’s robust capability of controlling the longitudinal and transverse positions of holograms under LCP and RCP waves independently.It’s expected that the proposed CBR-CM opens up ways of realizing more sophisticated and advanced devices with multiple independent information channels,which may provide technical assistance for digital EM environment reproduction.

Teaching Reform and Exploration of Practical Courses Based on Programmable Control

As the country continues to promote the development of intelligent manufacturing,all industries are carrying out enterprise automation upgrading,the Pearl River Delta Intelligent Manufacturing Conference held in March 2024 provides a direction guide for each enterprise on how to integrate the intelligent manufacturing technology into each link and provide direction guidance for enterprises to create new models and new business formats.College teachers,in focusing on the teaching process,should closely match the enterprise and social needs and cultivate excellent students.As the core controller of automation control,the application of programmable controllers in teaching is particularly important.In practical classes,by setting progressive difficulty,project guidance,team collaboration,and other links,students can master the automation equipment design of programmable control in repeated practice.

Finite-time Prescribed Performance Time-Varying Formation Control for Second-Order Multi-Agent Systems With Non-Strict Feedback Based on a Neural Network Observer

This paper studies the problem of time-varying formation control with finite-time prescribed performance for nonstrict feedback second-order multi-agent systems with unmeasured states and unknown nonlinearities.To eliminate nonlinearities,neural networks are applied to approximate the inherent dynamics of the system.In addition,due to the limitations of the actual working conditions,each follower agent can only obtain the locally measurable partial state information of the leader agent.To address this problem,a neural network state observer based on the leader state information is designed.Then,a finite-time prescribed performance adaptive output feedback control strategy is proposed by restricting the sliding mode surface to a prescribed region,which ensures that the closed-loop system has practical finite-time stability and that formation errors of the multi-agent systems converge to the prescribed performance bound in finite time.Finally,a numerical simulation is provided to demonstrate the practicality and effectiveness of the developed algorithm.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Vibration Control of A Flexible Marine Riser System Subject to Input Dead Zone and Extraneous Disturbances

An observer-based adaptive backstepping boundary control is proposed for vibration control of flexible offshore riser systems with unknown nonlinear input dead zone and uncertain environmental disturbances.The control algorithm can update the control law online through real-time data to make the controller adapt to the environment and improve the control precision.Specifically,based on the adaptive backstepping framework,virtual control laws and Lyapunov functions are designed for each subsystem.Three direction interference observers are designed to track the timevarying boundary disturbance.On this basis,the inverse of the dead zone and linear state transformation are used to compensate for the original system and eliminate the adverse effects of the dead zone.In addition,the stability of the closed-loop system is proven by Lyapunov stability theory.All the system states are bounded,and the vibration offset of the riser converges to a small area of the initial position.Finally,four examples of flexible marine risers are simulated in MATLAB to verify the effectiveness of the proposed controller.

Efficacy and predictive factors of transarterial chemoembolization combined with lenvatinib plus programmed cell death protein-1 inhibition for unresectable hepatocellular carcinoma

BACKGROUND The efficacy and safety of transarterial chemoembolization(TACE)combined with lenvatinib plus programmed cell death protein-1(PD-1)for unresectable hepato-cellular carcinoma(HCC)have rarely been evaluated and it is unknown which factors are related to efficacy.AIM To evaluate the efficacy and independent predictive factors of TACE combined with lenvatinib plus PD-1 inhibitors for unresectable HCC.METHODS This study retrospectively enrolled patients with unresectable HCC who received TACE/lenvatinib/PD-1 treatment between March 2019 and April 2022.Overall survival(OS)and progression-free survival(PFS)were determined.The objective response rate(ORR)and disease control rate(DCR)were evaluated in accordance with the modified Response Evaluation Criteria in Solid Tumors.Additionally,the prognostic factors affecting the clinical outcome were assessed.RESULTS One hundred and two patients were enrolled with a median follow-up duration of 12.63 months.The median OS was 26.43 months(95%CI:17.00-35.87),and the median PFS was 10.07 months(95%CI:8.50-11.65).The ORR and DCR were 61.76%and 81.37%,respectively.The patients with Barcelona Clinic Liver Cancer Classification(BCLC)B stage,early neutrophil-to-lymphocyte ratio(NLR)response(decrease),or early alpha-fetoprotein(AFP)response(decrease>20%)had superior OS and PFS than their counterparts.CONCLUSION This study showed that TACE/lenvatinib/PD-1 treatment was well tolerated with encouraging efficacy in patients with unresectable HCC.The patients with BCLC B-stage disease with early NLR response(decrease)and early AFP response(decrease>20%)may achieve better clinical outcomes with this triple therapy.

Control of highly pathogenic avian influenza through vaccination

The stamping-out strategy has been used to control highly pathogenic avian influenza viruses in many countries,driven by the belief that vaccination would not be successful against such viruses and fears that avian influenza virus in vaccinated birds would evolve more rapidly and pose a greater risk to humans.In this review,we summarize the successes in controlling highly pathogenic avian influenza in China and make suggestions regarding the requirements for vaccine selection and effectiveness.In addition,we present evidence that vaccination of poultry not only eliminates human infection with avian influenza virus,but also significantly reduces and abolishes some harmful characteristics of avian influenza virus.

An Integrated Control Framework for Torque Vectoring and Active Suspension System

Four-wheel independently driven electric vehicles(FWID-EV)endow a flexible and scalable control framework to improve vehicle performance.This paper integrates the torque vectoring and active suspension system(ASS)to enhance the vehicle’s longitudinal and vertical motion control performance.While the nonlinear characteristic of the tire model leads to a relatively heavier computational burden.To facilitate the controller design and ease the load,a half-vehicle dynamics system is built and simplified to the linear-time-varying(LTV)model.Then a model predictive controller is developed by formulating the objective function by comprehensively considering the safety,energy-saving and comfort requirements.The in-wheel motor efficiency and the power loss of tire slip are treated as optimization indices in this work to reduce energy consumption.Finally,the effectiveness of the proposed controller is verified through the rapid-control-prototype(RCP)test.The results demonstrate the enhancement of the energy-saving as well as comfort on the basis of vehicle stability.

Programmable Logic Controller Block Monitoring System for Memory Attack Defense in Industrial Control Systems

Cyberattacks targeting industrial control systems(ICS)are becoming more sophisticated and advanced than in the past.A programmable logic controller(PLC),a core component of ICS,controls and monitors sensors and actuators in the field.However,PLC has memory attack threats such as program injection and manipulation,which has long been a major target for attackers,and it is important to detect these attacks for ICS security.To detect PLC memory attacks,a security system is required to acquire and monitor PLC memory directly.In addition,the performance impact of the security system on the PLC makes it difficult to apply to the ICS.To address these challenges,this paper proposes a system to detect PLC memory attacks by continuously acquiring and monitoring PLC memory.The proposed system detects PLC memory attacks by acquiring the program blocks and block information directly from the same layer as the PLC and then comparing them in bytes with previous data.Experiments with Siemens S7-300 and S7-400 PLC were conducted to evaluate the PLC memory detection performance and performance impact on PLC.The experimental results demonstrate that the proposed system detects all malicious organization block(OB)injection and data block(DB)manipulation,and the increment of PLC cycle time,the impact on PLC performance,was less than 1 ms.The proposed system detects PLC memory attacks with a simpler detection method than earlier studies.Furthermore,the proposed system can be applied to ICS with a small performance impact on PLC.

Mitochondrial dysfunction and quality control lie at the heart of subarachnoid hemorrhage

The dramatic increase in intracranial pressure after subarachnoid hemorrhage leads to a decrease in cerebral perfusion pressure and a reduction in cerebral blood flow.Mitochondria are directly affected by direct factors such as ischemia,hypoxia,excitotoxicity,and toxicity of free hemoglobin and its degradation products,which trigger mitochondrial dysfunction.Dysfunctional mitochondria release large amounts of reactive oxygen species,inflammatory mediators,and apoptotic proteins that activate apoptotic pathways,further damaging cells.In response to this array of damage,cells have adopted multiple mitochondrial quality control mechanisms through evolution,including mitochondrial protein quality control,mitochondrial dynamics,mitophagy,mitochondrial biogenesis,and intercellular mitochondrial transfer,to maintain mitochondrial homeostasis under pathological conditions.Specific interventions targeting mitochondrial quality control mechanisms have emerged as promising therapeutic strategies for subarachnoid hemorrhage.This review provides an overview of recent research advances in mitochondrial pathophysiological processes after subarachnoid hemorrhage,particularly mitochondrial quality control mechanisms.It also presents potential therapeutic strategies to target mitochondrial quality control in subarachnoid hemorrhage.

A Blockchain-Based Access Control Scheme for Reputation Value Attributes of the Internet of Things

The Internet of Things(IoT)access controlmechanism may encounter security issues such as single point of failure and data tampering.To address these issues,a blockchain-based IoT reputation value attribute access control scheme is proposed.Firstly,writing the reputation value as an attribute into the access control policy,and then deploying the access control policy in the smart contract of the blockchain system can enable the system to provide more fine-grained access control;Secondly,storing a large amount of resources fromthe Internet of Things in Inter Planetary File System(IPFS)to improve system throughput;Finally,map resource access operations to qualification tokens to improve the performance of the access control system.Complete simulation experiments based on the Hyperledger Fabric platform.Fromthe simulation experimental results,it can be seen that the access control system can achieve more fine-grained and dynamic access control while maintaining high throughput and low time delay,providing sufficient reliability and security for access control of IoT devices.

A Review on Vibration Control of Multiple Cylinders Subjected to FlowInduced Vibrations

The fatigue damage caused by flow-induced vibration(FIV)is one of the major concerns for multiple cylindrical structures in many engineering applications.The FIV suppression is of great importance for the security of many cylindrical structures.Many active and passive control methods have been employed for the vibration suppression of an isolated cylinder undergoing vortex-induced vibrations(VIV).The FIV suppression methods are mainly extended to the multiple cylinders from the vibration control of the isolated cylinder.Due to the mutual interference between the multiple cylinders,the FIV mechanism is more complex than the VIV mechanism,which makes a great challenge for the FIV suppression.Some efforts have been devoted to vibration suppression of multiple cylinder systems undergoing FIV over the past two decades.The control methods,such as helical strakes,splitter plates,control rods and flexible sheets,are not always effective,depending on many influence factors,such as the spacing ratio,the arrangement geometrical shape,the flow velocity and the parameters of the vibration control devices.The FIV response,hydrodynamic features and wake patterns of the multiple cylinders equipped with vibration control devices are reviewed and summarized.The FIV suppression efficiency of the vibration control methods are analyzed and compared considering different influence factors.Further research on the FIV suppression of multiple cylinders is suggested to provide insight for the development of FIV control methods and promote engineering applications of FIV control methods.

Communication Resource-Efficient Vehicle Platooning Control With Various Spacing Policies

Platooning represents one of the key features that connected automated vehicles may possess as it allows multiple automated vehicles to be maneuvered cooperatively with small headways on roads. However, a critical challenge in accomplishing automated vehicle platoons is to deal with the effects of intermittent and sporadic vehicle-to-vehicle data transmissions caused by limited wireless communication resources. This paper addresses the co-design problem of dynamic event-triggered communication scheduling and cooperative adaptive cruise control for a convoy of automated vehicles with diverse spacing policies. The central aim is to achieve automated vehicle platooning under various gap references with desired platoon stability and spacing performance requirements, while simultaneously improving communication efficiency. Toward this aim, a dynamic event-triggered scheduling mechanism is developed such that the intervehicle data transmissions are scheduled dynamically and efficiently over time. Then, a tractable co-design criterion on the existence of both the admissible event-driven cooperative adaptive cruise control law and the desired scheduling mechanism is derived. Finally, comparative simulation results are presented to substantiate the effectiveness and merits of the obtained results.

A 16-bit 18-MSPS flash-assisted SAR ADC with hybrid synchronous and asynchronous control logic

This paper presents a 16-bit,18-MSPS(million samples per second)flash-assisted successive-approximation-register(SAR)analog-to-digital converter(ADC)utilizing hybrid synchronous and asynchronous(HYSAS)timing control logic based on an on-chip delay-locked loop(DLL).The HYSAS scheme can provide a longer settling time for the capacitive digital-to-analog converter(CDAC)than the synchronous and asynchronous SAR ADC.Therefore,the issue of incomplete settling or ringing in the DAC voltage for cases of either on-chip or off-chip reference voltage can be solved to a large extent.In addition,the fore-ground calibration of the CDAC’s mismatch is performed with a finite-impulse-response bandpass filter(FIR-BPF)based least-mean-square(LMS)algorithm in an off-chip FPGA(field programmable gate array).Fabricated in 40-nm CMOS process,the proto-type ADC achieves 94.02-dB spurious-free dynamic range(SFDR),and 75.98-dB signal-to-noise-and-distortion ratio(SNDR)for a 2.88-MHz input under 18-MSPS sampling rate.

Distributed Platooning Control of Automated Vehicles Subject to Replay Attacks Based on Proportional Integral Observers

Secure platooning control plays an important role in enhancing the cooperative driving safety of automated vehicles subject to various security vulnerabilities.This paper focuses on the distributed secure control issue of automated vehicles affected by replay attacks.A proportional-integral-observer(PIO)with predetermined forgetting parameters is first constructed to acquire the dynamical information of vehicles.Then,a time-varying parameter and two positive scalars are employed to describe the temporal behavior of replay attacks.In light of such a scheme and the common properties of Laplace matrices,the closed-loop system with PIO-based controllers is transformed into a switched and time-delayed one.Furthermore,some sufficient conditions are derived to achieve the desired platooning performance by the view of the Lyapunov stability theory.The controller gains are analytically determined by resorting to the solution of certain matrix inequalities only dependent on maximum and minimum eigenvalues of communication topologies.Finally,a simulation example is provided to illustrate the effectiveness of the proposed control strategy.

Cytokine release syndrome triggered by programmed death 1 blockade(sintilimab)therapy in a psoriasis patient:A case report

BACKGROUND In recent years,immune checkpoint inhibitors(ICIs)have demonstrated remarkable efficacy across diverse malignancies.Notably,in patients with advanced gastric cancer,the use of programmed death 1(PD-1)blockade has significantly prolonged overall survival,marking a pivotal advancement comparable to the impact of Herceptin over the past two decades.While the therapeutic benefits of ICIs are evident,the increasing use of immunotherapy has led to an increase in immune-related adverse events.CASE SUMMARY This article presents the case of a patient with advanced gastric cancer and chronic plaque psoriasis.Following sintilimab therapy,the patient developed severe rashes accompanied by cytokine release syndrome(CRS).Fortunately,effective management was achieved through the administration of glucocorticoid,tocilizumab,and acitretin,which resulted in favorable outcomes.CONCLUSION Glucocorticoid and tocilizumab therapy was effective in managing CRS after PD-1 blockade therapy for gastric cancer in a patient with chronic plaque psoriasis.

Path Planning and Tracking Control for Parking via Soft Actor-Critic Under Non-Ideal Scenarios

Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.