Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

基于DMC策略的高速铁路受电弓系统机械振动研究

高速铁路受电弓系统要求电力机车在运行时保证弓网接触的稳定和安全,特点是弓网电流传输平稳,参数实时监测存储,弓网机械振动误差小和运行曲线调节速度快.本文首先分析了高速受电弓系统运行参数的设计指标和曲线特点,提出弓网机械振动的结构;其次,分析了弓网机械振动的原理和DMC参数预测模型,建立基于开环预估控制算法流程图.DMC作为一种类聚优化曲线的开环控制,其主要工作原理是运用受电弓系统动态的参数指标为参考输入,以弓网机械振动在线检测和曲线调整为主的优化策略.通过算法与系统性能之间的定量关系,在应用中对设计参数进行调试.实验结果表明,基于DMC开环控制方法优于GPC闭环反馈控制效果,且具有最优解.

Vibration Control of A Flexible Marine Riser System Subject to Input Dead Zone and Extraneous Disturbances

An observer-based adaptive backstepping boundary control is proposed for vibration control of flexible offshore riser systems with unknown nonlinear input dead zone and uncertain environmental disturbances.The control algorithm can update the control law online through real-time data to make the controller adapt to the environment and improve the control precision.Specifically,based on the adaptive backstepping framework,virtual control laws and Lyapunov functions are designed for each subsystem.Three direction interference observers are designed to track the timevarying boundary disturbance.On this basis,the inverse of the dead zone and linear state transformation are used to compensate for the original system and eliminate the adverse effects of the dead zone.In addition,the stability of the closed-loop system is proven by Lyapunov stability theory.All the system states are bounded,and the vibration offset of the riser converges to a small area of the initial position.Finally,four examples of flexible marine risers are simulated in MATLAB to verify the effectiveness of the proposed controller.

Control of highly pathogenic avian influenza through vaccination

The stamping-out strategy has been used to control highly pathogenic avian influenza viruses in many countries,driven by the belief that vaccination would not be successful against such viruses and fears that avian influenza virus in vaccinated birds would evolve more rapidly and pose a greater risk to humans.In this review,we summarize the successes in controlling highly pathogenic avian influenza in China and make suggestions regarding the requirements for vaccine selection and effectiveness.In addition,we present evidence that vaccination of poultry not only eliminates human infection with avian influenza virus,but also significantly reduces and abolishes some harmful characteristics of avian influenza virus.

A Blockchain-Based Access Control Scheme for Reputation Value Attributes of the Internet of Things

The Internet of Things(IoT)access controlmechanism may encounter security issues such as single point of failure and data tampering.To address these issues,a blockchain-based IoT reputation value attribute access control scheme is proposed.Firstly,writing the reputation value as an attribute into the access control policy,and then deploying the access control policy in the smart contract of the blockchain system can enable the system to provide more fine-grained access control;Secondly,storing a large amount of resources fromthe Internet of Things in Inter Planetary File System(IPFS)to improve system throughput;Finally,map resource access operations to qualification tokens to improve the performance of the access control system.Complete simulation experiments based on the Hyperledger Fabric platform.Fromthe simulation experimental results,it can be seen that the access control system can achieve more fine-grained and dynamic access control while maintaining high throughput and low time delay,providing sufficient reliability and security for access control of IoT devices.

A Review on Vibration Control of Multiple Cylinders Subjected to FlowInduced Vibrations

The fatigue damage caused by flow-induced vibration(FIV)is one of the major concerns for multiple cylindrical structures in many engineering applications.The FIV suppression is of great importance for the security of many cylindrical structures.Many active and passive control methods have been employed for the vibration suppression of an isolated cylinder undergoing vortex-induced vibrations(VIV).The FIV suppression methods are mainly extended to the multiple cylinders from the vibration control of the isolated cylinder.Due to the mutual interference between the multiple cylinders,the FIV mechanism is more complex than the VIV mechanism,which makes a great challenge for the FIV suppression.Some efforts have been devoted to vibration suppression of multiple cylinder systems undergoing FIV over the past two decades.The control methods,such as helical strakes,splitter plates,control rods and flexible sheets,are not always effective,depending on many influence factors,such as the spacing ratio,the arrangement geometrical shape,the flow velocity and the parameters of the vibration control devices.The FIV response,hydrodynamic features and wake patterns of the multiple cylinders equipped with vibration control devices are reviewed and summarized.The FIV suppression efficiency of the vibration control methods are analyzed and compared considering different influence factors.Further research on the FIV suppression of multiple cylinders is suggested to provide insight for the development of FIV control methods and promote engineering applications of FIV control methods.

基于DMC算法的精馏塔精馏段温度预测-PID双层控制系统应用分析

以精馏塔精馏段温度控制为例,提出基于动态矩阵控制(DMC)算法的预测-PID双层控制方案,并通过MATLAB/Simulink联合仿真方法,对PID-PID、预测-PID两种双层控制系统进行仿真对比分析,得出预测-PID双层控制系统具有跟踪性能好、超调量小、调节时间短且抗干扰能力强的优点。

A 16-bit 18-MSPS flash-assisted SAR ADC with hybrid synchronous and asynchronous control logic

This paper presents a 16-bit,18-MSPS(million samples per second)flash-assisted successive-approximation-register(SAR)analog-to-digital converter(ADC)utilizing hybrid synchronous and asynchronous(HYSAS)timing control logic based on an on-chip delay-locked loop(DLL).The HYSAS scheme can provide a longer settling time for the capacitive digital-to-analog converter(CDAC)than the synchronous and asynchronous SAR ADC.Therefore,the issue of incomplete settling or ringing in the DAC voltage for cases of either on-chip or off-chip reference voltage can be solved to a large extent.In addition,the fore-ground calibration of the CDAC’s mismatch is performed with a finite-impulse-response bandpass filter(FIR-BPF)based least-mean-square(LMS)algorithm in an off-chip FPGA(field programmable gate array).Fabricated in 40-nm CMOS process,the proto-type ADC achieves 94.02-dB spurious-free dynamic range(SFDR),and 75.98-dB signal-to-noise-and-distortion ratio(SNDR)for a 2.88-MHz input under 18-MSPS sampling rate.

Mitochondrial dysfunction and quality control lie at the heart of subarachnoid hemorrhage

The dramatic increase in intracranial pressure after subarachnoid hemorrhage leads to a decrease in cerebral perfusion pressure and a reduction in cerebral blood flow.Mitochondria are directly affected by direct factors such as ischemia,hypoxia,excitotoxicity,and toxicity of free hemoglobin and its degradation products,which trigger mitochondrial dysfunction.Dysfunctional mitochondria release large amounts of reactive oxygen species,inflammatory mediators,and apoptotic proteins that activate apoptotic pathways,further damaging cells.In response to this array of damage,cells have adopted multiple mitochondrial quality control mechanisms through evolution,including mitochondrial protein quality control,mitochondrial dynamics,mitophagy,mitochondrial biogenesis,and intercellular mitochondrial transfer,to maintain mitochondrial homeostasis under pathological conditions.Specific interventions targeting mitochondrial quality control mechanisms have emerged as promising therapeutic strategies for subarachnoid hemorrhage.This review provides an overview of recent research advances in mitochondrial pathophysiological processes after subarachnoid hemorrhage,particularly mitochondrial quality control mechanisms.It also presents potential therapeutic strategies to target mitochondrial quality control in subarachnoid hemorrhage.

DMC模型视域下学前教育APP游戏设计策略研究——以宝宝巴士世界APP为例

目的:随着计算机技术和智能移动设备的发展,开发商们纷纷通过游戏的方式介入儿童教育。但蓬勃发展的国内学前教育APP也暴露出一系列问题,根本原因是游戏化学习的设计实践欠缺理论指导,使得设计者在游戏中难以平衡教育性和娱乐性。文章探讨DMC模型视域下学前教育APP游戏设计策略,旨在为“游戏化+学前教育”的实践提供参考,推动教育领域持续创新与发展。方法:基于游戏化学习理论,以凯文·韦巴赫的DMC模型为分析框架,针对经典学前教育APP,深入分析其游戏化元素的结构和应用效果,从组建层、机制层和动力层三个层面构建游戏设计策略。结果:学前教育APP的游戏化设计须考虑多方面因素:内容型组件应少用文本,统一视觉风格并控制信息承载量;交互型组件须考虑儿童发育特征,设计适宜的交互手势和图标尺寸;机制层的反馈元素应及时、易懂且具有激励性;动力层则以情感元素为主,以帮助孩子成长为目的,并结合家长控制功能,约束孩子的行为,确保游戏环境的安全性和教育效果。结论:未来的研究可进一步探讨教育游戏的内在逻辑,并结合不同教育内容和用户群体应用DMC模型,以优化游戏化学习的实践和提高其设计的适用性。

Communication Resource-Efficient Vehicle Platooning Control With Various Spacing Policies

Platooning represents one of the key features that connected automated vehicles may possess as it allows multiple automated vehicles to be maneuvered cooperatively with small headways on roads. However, a critical challenge in accomplishing automated vehicle platoons is to deal with the effects of intermittent and sporadic vehicle-to-vehicle data transmissions caused by limited wireless communication resources. This paper addresses the co-design problem of dynamic event-triggered communication scheduling and cooperative adaptive cruise control for a convoy of automated vehicles with diverse spacing policies. The central aim is to achieve automated vehicle platooning under various gap references with desired platoon stability and spacing performance requirements, while simultaneously improving communication efficiency. Toward this aim, a dynamic event-triggered scheduling mechanism is developed such that the intervehicle data transmissions are scheduled dynamically and efficiently over time. Then, a tractable co-design criterion on the existence of both the admissible event-driven cooperative adaptive cruise control law and the desired scheduling mechanism is derived. Finally, comparative simulation results are presented to substantiate the effectiveness and merits of the obtained results.

Path Planning and Tracking Control for Parking via Soft Actor-Critic Under Non-Ideal Scenarios

Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.

A Tutorial on Quantized Feedback Control

In this tutorial paper, we explore the field of quantized feedback control, which has gained significant attention due to the growing prevalence of networked control systems. These systems require the transmission of feedback information, such as measurements and control signals, over digital networks, presenting novel challenges in estimation and control design. Our examination encompasses various topics, including the minimal information needed for effective feedback control, the design of quantizers, strategies for quantized control design and estimation,achieving consensus control with quantized data, and the pursuit of high-precision tracking using quantized measurements.

Risk-Informed Model-Free Safe Control of Linear Parameter-Varying

This paper presents a risk-informed data-driven safe control design approach for a class of stochastic uncertain nonlinear discrete-time systems.The nonlinear system is modeled using linear parameter-varying(LPV)systems.A model-based probabilistic safe controller is first designed to guarantee probabilisticλ-contractivity(i.e.,stability and invariance)of the LPV system with respect to a given polyhedral safe set.To obviate the requirement of knowing the LPV system model and to bypass identifying its open-loop model,its closed-loop data-based representation is provided in terms of state and scheduling data as well as a decision variable.It is shown that the variance of the closedloop system,as well as the probability of safety satisfaction,depends on the decision variable and the noise covariance.A minimum-variance direct data-driven gain-scheduling safe control design approach is presented next by designing the decision variable such that all possible closed-loop system realizations satisfy safety with the highest confidence level.This minimum-variance approach is a control-oriented learning method since it minimizes the variance of the state of the closed-loop system with respect to the safe set,and thus minimizes the risk of safety violation.Unlike the certainty-equivalent approach that results in a risk-neutral control design,the minimum-variance method leads to a risk-averse control design.It is shown that the presented direct risk-averse learning approach requires weaker data richness conditions than existing indirect learning methods based on system identification and can lead to a lower risk of safety violation.Two simulation examples along with an experimental validation on an autonomous vehicle are provided to show the effectiveness of the presented approach.

Finite-time Prescribed Performance Time-Varying Formation Control for Second-Order Multi-Agent Systems With Non-Strict Feedback Based on a Neural Network Observer

This paper studies the problem of time-varying formation control with finite-time prescribed performance for nonstrict feedback second-order multi-agent systems with unmeasured states and unknown nonlinearities.To eliminate nonlinearities,neural networks are applied to approximate the inherent dynamics of the system.In addition,due to the limitations of the actual working conditions,each follower agent can only obtain the locally measurable partial state information of the leader agent.To address this problem,a neural network state observer based on the leader state information is designed.Then,a finite-time prescribed performance adaptive output feedback control strategy is proposed by restricting the sliding mode surface to a prescribed region,which ensures that the closed-loop system has practical finite-time stability and that formation errors of the multi-agent systems converge to the prescribed performance bound in finite time.Finally,a numerical simulation is provided to demonstrate the practicality and effectiveness of the developed algorithm.

Quantization and Event-Triggered Policy Design for Encrypted Networked Control

This paper proposes a novel event-driven encrypted control framework for linear networked control systems(NCSs),which relies on two modified uniform quantization policies,the Paillier cryptosystem,and an event-triggered strategy.Due to the fact that only integers can work in the Pailler cryptosystem,both the real-valued control gain and system state need to be first quantized before encryption.This is dramatically different from the existing quantized control methods,where only the quantization of a single value,e.g.,the control input or the system state,is considered.To handle this issue,static and dynamic quantization policies are presented,which achieve the desired integer conversions and guarantee asymptotic convergence of the quantized system state to the equilibrium.Then,the quantized system state is encrypted and sent to the controller when the triggering condition,specified by a state-based event-triggered strategy,is satisfied.By doing so,not only the security and confidentiality of data transmitted over the communication network are protected,but also the ciphertext expansion phenomenon can be relieved.Additionally,by tactfully designing the quantization sensitivities and triggering error,the proposed event-driven encrypted control framework ensures the asymptotic stability of the overall closedloop system.Finally,a simulation example of the secure motion control for an inverted pendulum cart system is presented to evaluate the effectiveness of the theoretical results.

Practical Prescribed Time Tracking Control With Bounded Time-Varying Gain Under Non-Vanishing Uncertainties

This paper investigates the prescribed-time control(PTC) problem for a class of strict-feedback systems subject to non-vanishing uncertainties. The coexistence of mismatched uncertainties and non-vanishing disturbances makes PTC synthesis nontrivial. In this work, a control method that does not involve infinite time-varying gain is proposed, leading to a practical and global prescribed time tracking control solution for the strict-feedback systems, in spite of both the mismatched and nonvanishing uncertainties. Different from methods based on control switching to avoid the issue of infinite control gain that involves control discontinuity at the switching point, in our method a softening unit is exclusively included to ensure the continuity of the control action. Furthermore, in contrast to most existing prescribed-time control works where the control scheme is only valid on a finite time interval, in this work, the proposed control scheme is valid on the entire time interval. In addition, the prior information on the upper or lower bound of gi is not in need,enlarging the applicability of the proposed method. Both the theoretical analysis and numerical simulation confirm the effectiveness of the proposed control algorithm.

Noise-Tolerant ZNN-Based Data-Driven Iterative Learning Control for Discrete Nonaffine Nonlinear MIMO Repetitive Systems

Aiming at the tracking problem of a class of discrete nonaffine nonlinear multi-input multi-output(MIMO) repetitive systems subjected to separable and nonseparable disturbances, a novel data-driven iterative learning control(ILC) scheme based on the zeroing neural networks(ZNNs) is proposed. First, the equivalent dynamic linearization data model is obtained by means of dynamic linearization technology, which exists theoretically in the iteration domain. Then, the iterative extended state observer(IESO) is developed to estimate the disturbance and the coupling between systems, and the decoupled dynamic linearization model is obtained for the purpose of controller synthesis. To solve the zero-seeking tracking problem with inherent tolerance of noise,an ILC based on noise-tolerant modified ZNN is proposed. The strict assumptions imposed on the initialization conditions of each iteration in the existing ILC methods can be absolutely removed with our method. In addition, theoretical analysis indicates that the modified ZNN can converge to the exact solution of the zero-seeking tracking problem. Finally, a generalized example and an application-oriented example are presented to verify the effectiveness and superiority of the proposed process.

Quaternion-Based Adaptive Trajectory Tracking Control of a Rotor-Missile with Unknown Parameters Identification

This paper investigates the adaptive trajectory tracking control problem and the unknown parameter identification problem of a class of rotor-missiles with parametric system uncertainties.First,considering the uncertainty of structural and aerodynamic parameters,the six-degree-of-freedom(6Do F) nonlinear equations describing the position and attitude dynamics of the rotor-missile are established,respectively,in the inertial and body-fixed reference frames.Next,a hierarchical adaptive trajectory tracking controller that can guarantee closed-loop stability is proposed according to the cascade characteristics of the 6Do F dynamics.Then,a memory-augmented update rule of unknown parameters is proposed by integrating all historical data of the regression matrix.As long as the finitely excited condition is satisfied,the precise identification of unknown parameters can be achieved.Finally,the validity of the proposed trajectory tracking controller and the parameter identification method is proved through Lyapunov stability theory and numerical simulations.

基于DMC的电厂脱硫pH值优化控制研究

烟气脱硫控制系统中,吸收塔浆液的pH值对系统的脱硫效率至关重要。针对湿法烟气脱硫(WFGD)过程中浆液pH值变化的非线性及其存在较大的滞后性问题,设计一种基于动态矩阵控制(DMC)算法的控制方法,以实现对浆液pH值的精确跟踪控制。结果表明,与传统PID控制算法相比,DMC算法对浆液pH值的设定输入和波动变化具有更强的跟踪能力和更高的控制精度。该方法可以进一步满足系统的实时控制要求。此外,DMC算法还可有效抑制WFGD过程中的随机干扰,提升系统的稳定性和抗干扰能力。