Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

Dynamic adaptive spatio-temporal graph network for COVID-19 forecasting

Appropriately characterising the mixed space-time relations of the contagion process caused by hybrid space and time factors remains the primary challenge in COVID-19 forecasting.However,in previous deep learning models for epidemic forecasting,spatial and temporal variations are captured separately.A unified model is developed to cover all spatio-temporal relations.However,this measure is insufficient for modelling the complex spatio-temporal relations of infectious disease transmission.A dynamic adaptive spatio-temporal graph network(DASTGN)is proposed based on attention mechanisms to improve prediction accuracy.In DASTGN,complex spatio-temporal relations are depicted by adaptively fusing the mixed space-time effects and dynamic space-time dependency structure.This dual-scale model considers the time-specific,space-specific,and direct effects of the propagation process at the fine-grained level.Furthermore,the model characterises impacts from various space-time neighbour blocks under time-varying interventions at the coarse-grained level.The performance comparisons on the three COVID-19 datasets reveal that DASTGN achieves state-of-the-art results with a maximum improvement of 17.092%in the root mean-square error and 11.563%in the mean absolute error.Experimental results indicate that the mechanisms of designing DASTGN can effectively detect some spreading characteristics of COVID-19.The spatio-temporal weight matrices learned in each proposed module reveal diffusion patterns in various scenarios.In conclusion,DASTGN has successfully captured the dynamic spatio-temporal variations of COVID-19,and considering multiple dynamic space-time relationships is essential in epidemic forecasting.

Inversion of Seabed Geotechnical Properties in the Arctic Chukchi Deep Sea Basin Based on Time Domain Adaptive Search Matching Algorithm

The chirp sub-bottom profiler,for its high resolution,easy accessibility and cost-effectiveness,has been widely used in acoustic detection.In this paper,the acoustic impedance and grain size compositions were obtained based on the chirp sub-bottom profiler data collected in the Chukchi Plateau area during the 11th Arctic Expedition of China.The time-domain adaptive search matching algorithm was used and validated on our established theoretical model.The misfit between the inversion result and the theoretical model is less than 0.067%.The grain size was calculated according to the empirical relationship between the acoustic impedance and the grain size of the sediment.The average acoustic impedance of sub-seafloor strata is 2.5026×10^(6) kg(s m^(2))^(-1)and the average grain size(θvalue)of the seafloor surface sediment is 7.1498,indicating the predominant occurrence of very fine silt sediment in the study area.Comparison of the inversion results and the laboratory measurements of nearby borehole samples shows that they are in general agreement.

Time/Space and Inertial System Reconsidered Based on the Adaptive Dynamical View

We have succeeded in 2-slit interference simulation by assuming that a travelling particle interacts with its environment, getting information on the environmental condition according to the adaptive dynamics by Ohya, thus proposed the possibility that the entanglement comes from the interaction with the environment (Ando et al., 2023). This concept means that there should be no isolated or inertial system other than our unique universe space. Taking this message into account and assuming that the signal velocity is constant against our unique universe space, we reconsidered the inertial system and relativity theory by Galilei and Einstein and found several misunderstandings and errors. Time delay and Lorentz shrinkage were derived similarly to the prediction by special relativity theory, but Lorentz transformation and 4-dimensional time/space view were not. They must have implicitly and unconsciously assumed that any signals transfer information without giving any influences to any systems different from our adaptive dynamical view. We propose that their relativity theories should be reinterpreted in view of adaptive dynamics.

Refinement of Adaptive Dynamical Simulation of Quantum Mechanical Double Slit Interference Phenomenon

We applied adaptive dynamics to double slit interference phenomenon using particle model and obtained partial successful results in our previous report. The patterns qualitatively corresponded well with experiments. Several properties such as concave single slit pattern and large influence of slight displacement of the emission position were different from the experimental results. In this study we tried other slit conditions and obtained consistent patterns with experiments. We do not claim that the adaptive dynamics is the principle of quantum mechanics, but the present results support the probability of adaptive dynamics as the candidate of the basis of quantum mechanics. We discuss the advantages of the adaptive dynamical view for foundations of quantum mechanics.

Adaptive Method’s Application for the Identification Systems in Dynamic Weighing

In this work,the possibility of adaptive algorithm in WIM(weight-in-motion)systems,in which fibre optic sensors are used,is shown.Appointment of dynamic weighing device consists in determining the weight and type of vehicle.In this work an algorithm for processing the input data and fiber optic sensor to create the database used in the algorithm is presented.The results of the algorithm for the identification of vehicles are given.The conclusions are made and options of increasing the accuracy of the identification algorithm are considered.

Distributed Adaptive Resource Allocation:An Uncertain Saddle-Point Dynamics Viewpoint

This paper addresses distributed adaptive optimal resource allocation problems over weight-balanced digraphs.By leveraging state-of-the-art adaptive coupling designs for multiagent systems,two adaptive algorithms are proposed,namely a directed-spanning-tree-based algorithm and a node-based algorithm.The benefits of these algorithms are that they require neither sufficiently small or unitary step sizes,nor global knowledge of Laplacian eigenvalues,which are widely required in the literature.It is shown that both algorithms belong to a class of uncertain saddle-point dynamics,which can be tackled by repeatedly adopting the Peter-Paul inequality in the framework of Lyapunov theory.Thanks to this new viewpoint,global asymptotic convergence of both algorithms can be proven in a unified way.The effectiveness of the proposed algorithms is validated through numerical simulations and case studies in IEEE 30-bus and 118-bus power systems.

Hierarchical adaptive stereo matching algorithm for obstacle detection with dynamic programming

An adaptive weighted stereo matching algorithm with multilevel and bidirectional dynamic programming based on ground control points （GCPs） is presented. To decrease time complexity without losing matching precision, using a multilevel search scheme, the coarse matching is processed in typical disparity space image, while the fine matching is processed in disparity-offset space image. In the upper level, GCPs are obtained by enhanced volumetric iterative algorithm enforcing the mutual constraint and the threshold constraint. Under the supervision of the highly reliable GCPs, bidirectional dynamic programming framework is employed to solve the inconsistency in the optimization path. In the lower level, to reduce running time, disparity-offset space is proposed to efficiently achieve the dense disparity image. In addition, an adaptive dual support-weight strategy is presented to aggregate matching cost, which considers photometric and geometric information. Further, post-processing algorithm can ameliorate disparity results in areas with depth discontinuities and related by occlusions using dual threshold algorithm, where missing stereo information is substituted from surrounding regions. To demonstrate the effectiveness of the algorithm, we present the two groups of experimental results for four widely used standard stereo data sets, including discussion on performance and comparison with other methods, which show that the algorithm has not only a fast speed, but also significantly improves the efficiency of holistic optimization.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Value Iteration-Based Cooperative Adaptive Optimal Control for Multi-Player Differential Games With Incomplete Information

This paper presents a novel cooperative value iteration(VI)-based adaptive dynamic programming method for multi-player differential game models with a convergence proof.The players are divided into two groups in the learning process and adapt their policies sequentially.Our method removes the dependence of admissible initial policies,which is one of the main drawbacks of the PI-based frameworks.Furthermore,this algorithm enables the players to adapt their control policies without full knowledge of others’ system parameters or control laws.The efficacy of our method is illustrated by three examples.

Dynamic Constraint-Driven Event-Triggered Control of Strict-Feedback Systems Without Max/Min Values on Irregular Constraints

This work proposes an event-triggered adaptive control approach for a class of uncertain nonlinear systems under irregular constraints.Unlike the constraints considered in most existing papers,here the external irregular constraints are considered and a constraints switching mechanism(CSM)is introduced to circumvent the difficulties arising from irregular output constraints.Based on the CSM,a new class of generalized barrier functions are constructed,which allows the control results to be independent of the maximum and minimum values(MMVs)of constraints and thus extends the existing results.Finally,we proposed a novel dynamic constraint-driven event-triggered strategy(DCDETS),under which the stress on signal transmission is reduced greatly and no constraints are violated by making a dynamic trade-off among system state,external constraints,and inter-execution intervals.It is proved that the system output is driven to close to the reference trajectory and the semi-global stability is guaranteed under the proposed control scheme,regardless of the external irregular output constraints.Simulation also verifies the effectiveness and benefits of the proposed method.

Model-driven full system dynamics estimation of PMSM-driven chain shell magazine

Based on the system dynamic model, a full system dynamics estimation method is proposed for a chain shell magazine driven by a permanent magnet synchronous motor(PMSM). An adaptive extended state observer(AESO) is proposed to estimate the unmeasured states and disturbance, in which the model parameters are adjusted in real time. Theoretical analysis shows that the estimation errors of the disturbances and unmeasured states converge exponentially to zero, and the parameter estimation error can be obtained from the extended state. Then, based on the extended state of the AESO, a novel parameter estimation law is designed. Due to the convergence of AESO, the novel parameter estimation law is insensitive to controllers and excitation signal. Under persistent excitation(PE) condition, the estimated parameters will converge to a compact set around the actual parameter value. Without PE signal, the estimated parameters will converge to zero for the extended state. Simulation and experimental results show that the proposed method can accurately estimate the unmeasured states and disturbance of the chain shell magazine, and the estimated parameters will converge to the actual value without strictly continuous PE signals.

Robust adaptive dynamic surface control for nonlinear uncertain systems

We propose a new method for robust adaptive backstepping control of nonlinear systems with parametric uncertainties and disturbances in the strict feedback form. The method is called dynamic surface control. Traditional backstepping algorithms require repeated differentiations of the modelled nonlinearities. The addition of n first order low pass filters allows the algorithm to be implemented without differentiating any model nonlinearities, thus ending the complexity arising due to the 'explosion of terms' that makes other methods difficult to implement in practice. The combined robust adaptive backstepping/first order filter system is proved to be semiglobally asymptotically stable for sufficiently fast filters by a singular perturbation approach. The simulation results demonstrate the feasibility and effectiveness of the controller designed by the method.

Novel disturbance compensating dynamic positioning of dredgers based on adaptive backstepping

In order to deal with the dynamic positioning system control problems of dredgers working under strong dredging reaction or harsh environments,an adaptive backstepping method is proposed.Disturbances are estimated and compensated for by the adaptive method without extra sensors on dredging equipment,and the control mechanism is simplified.Adaptive control is used to compensate for the reaction and environmental disturbances on the dredger,so the dredger can maintain the desired position with a minimum error and shock.The proposed adaptive robust controller guarantees the global asymptotic stability of the closed-loop system and rapid position tracking of the dredger.The simulation results show that the proposed controller has superior performance in position tracking and robustness to large disturbances.

Residential Energy Scheduling for Variable Weather Solar Energy Based on Adaptive Dynamic Programming

The residential energy scheduling of solar energy is an important research area of smart grid. On the demand side, factors such as household loads, storage batteries, the outside public utility grid and renewable energy resources, are combined together as a nonlinear, time-varying, indefinite and complex system, which is difficult to manage or optimize. Many nations have already applied the residential real-time pricing to balance the burden on their grid. In order to enhance electricity efficiency of the residential micro grid, this paper presents an action dependent heuristic dynamic programming(ADHDP) method to solve the residential energy scheduling problem. The highlights of this paper are listed below. First,the weather-type classification is adopted to establish three types of programming models based on the features of the solar energy. In addition, the priorities of different energy resources are set to reduce the loss of electrical energy transmissions.Second, three ADHDP-based neural networks, which can update themselves during applications, are designed to manage the flows of electricity. Third, simulation results show that the proposed scheduling method has effectively reduced the total electricity cost and improved load balancing process. The comparison with the particle swarm optimization algorithm further proves that the present method has a promising effect on energy management to save cost.

Dynamics and adaptive control of a dual-arm space robot with closed-loop constraints and uncertain inertial parameters

A dynamics-based adaptive control approach is proposed for a planar dual-arm space robot in the presence of closed-loop constraints and uncertain inertial parameters of the payload. The controller is capable of controlling the po- sition and attitude of both the satellite base and the payload grasped by the manipulator end effectors. The equations of motion in reduced-order form for the constrained system are derived by incorporating the constraint equations in terms of accelerations into Kane＇s equations of the unconstrained system. Model analysis shows that the resulting equations perfectly meet the requirement of adaptive controller design. Consequently, by using an indirect approach, an adaptive control scheme is proposed to accomplish position/attitude trajectory tracking control with the uncertain parameters be- ing estimated on-line. The actuator redundancy due to the closed-loop constraints is utilized to minimize a weighted norm of the joint torques. Global asymptotic stability is proven by using Lyapunov＇s method, and simulation results are also presented to demonstrate the effectiveness of the proposed approach.

Adaptive Dynamic Surface Control for Integrated Missile Guidance and Autopilot

Integrated guidance and control for homing missiles utilizing adaptive dynamic surface control approach is considered based on the three channels independence design idea. A time-varying integrated guidance and control model with unmatched uncertainties is first formulated for the pitch channel, and an adaptive dynamic surface control algorithm is further developed to deal with these unmatched uncertainties. It is proved that the proposed feedback controller can ensure not only the accuracy of target interception, but also the stability of the missile dynamics. Then, the same control approach is further applied to the control design of the yaw and roll channels. The 6-degree-of-freedom （6-DOF） nonlinear missile simulation results demonstrate the feasibility and advantage of the proposed integrated guidance and control design scheme.

Inertia Matching During Dynamic Simulation of Vehicle Drive System on Test Bench

Framework and basic parameters of a test bench for motor drive system of electric vehicle (EV) are illuminated. Two kinds of electric drive models, one was for the electric vehicle drived on real road, the other was for that on test bench, are put forward. Then, dynamic analysis of these models is made in detail. Inertia matching method of the test bench is researched and some useful formulas and graphs are brought forward. The experiment of an electric bus is introduced in order to explain the usage of this inertia matching method.

Impact dynamics analysis of free-floating space manipulator capturing satellite on orbit and robust adaptive compound control algorithm design for suppressing motion

The impact dynamics, impact effect, and post-impact unstable motion sup- pression of free-floating space manipulator capturing a satellite on orbit are analyzed. Firstly, the dynamics equation of free-floating space manipulator is derived using the sec- ond Lagrangian equation. Combining the momentum conservation principle, the impact dynamics and effect between the space manipulator end-effector and satellite of the cap- ture process are analyzed with the momentum impulse method. Focusing on the unstable motion of space manipulator due to the above impact effect, a robust adaptive compound control algorithm is designed to suppress the above unstable motion. There is no need to control the free-floating base position to save the jet fuel. Finally, the simulation is proposed to show the impact effect and verify the validity of the control algorithm.