Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

Approximate Dynamic Programming for Stochastic Resource Allocation Problems

A stochastic resource allocation model, based on the principles of Markov decision processes(MDPs), is proposed in this paper. In particular, a general-purpose framework is developed, which takes into account resource requests for both instant and future needs. The considered framework can handle two types of reservations(i.e., specified and unspecified time interval reservation requests), and implement an overbooking business strategy to further increase business revenues. The resulting dynamic pricing problems can be regarded as sequential decision-making problems under uncertainty, which is solved by means of stochastic dynamic programming(DP) based algorithms. In this regard, Bellman’s backward principle of optimality is exploited in order to provide all the implementation mechanisms for the proposed reservation pricing algorithm. The curse of dimensionality, as the inevitable issue of the DP both for instant resource requests and future resource reservations,occurs. In particular, an approximate dynamic programming(ADP) technique based on linear function approximations is applied to solve such scalability issues. Several examples are provided to show the effectiveness of the proposed approach.

Using approximate dynamic programming for multi-ESM scheduling to track ground moving targets

This paper researches the adaptive scheduling problem of multiple electronic support measures(multi-ESM) in a ground moving radar targets tracking application. It is a sequential decision-making problem in uncertain environment. For adaptive selection of appropriate ESMs, we generalize an approximate dynamic programming(ADP) framework to the dynamic case. We define the environment model and agent model, respectively. To handle the partially observable challenge, we apply the unsented Kalman filter(UKF) algorithm for belief state estimation. To reduce the computational burden, a simulation-based approach rollout with a redesigned base policy is proposed to approximate the long-term cumulative reward. Meanwhile, Monte Carlo sampling is combined into the rollout to estimate the expectation of the rewards. The experiments indicate that our method outperforms other strategies due to its better performance in larger-scale problems.

Approximate Dynamic Programming for Self-Learning Control

This paper introduces a self-learning control approach based on approximate dynamic programming. Dynamic programming was introduced by Bellman in the 1950's for solving optimal control problems of nonlinear dynamical systems. Due to its high computational complexity, the applications of dynamic programming have been limited to simple and small problems. The key step in finding approximate solutions to dynamic programming is to estimate the performance index in dynamic programming. The optimal control signal can then be determined by minimizing (or maximizing) the performance index. Artificial neural networks are very efficient tools in representing the performance index in dynamic programming. This paper assumes the use of neural networks for estimating the performance index in dynamic programming and for generating optimal control signals, thus to achieve optimal control through self-learning.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

A novel stable value iteration-based approximate dynamic programming algorithm for discrete-time nonlinear systems

The convergence and stability of a value-iteration-based adaptive dynamic programming （ADP） algorithm are con- sidered for discrete-time nonlinear systems accompanied by a discounted quadric performance index. More importantly than sufficing to achieve a good approximate structure, the iterative feedback control law must guarantee the closed-loop stability. Specifically, it is firstly proved that the iterative value function sequence will precisely converge to the optimum. Secondly, the necessary and sufficient condition of the optimal value function serving as a Lyapunov function is investi- gated. We prove that for the case of infinite horizon, there exists a finite horizon length of which the iterative feedback control law will provide stability, and this increases the practicability of the proposed value iteration algorithm. Neural networks （NNs） are employed to approximate the value functions and the optimal feedback control laws, and the approach allows the implementation of the algorithm without knowing the internal dynamics of the system. Finally, a simulation example is employed to demonstrate the effectiveness of the developed optimal control method.

Two-stage robust optimization of power cost minimization problem in gunbarrel natural gas networks by approximate dynamic programming

In short-term operation of natural gas network,the impact of demand uncertainty is not negligible.To address this issue we propose a two-stage robust model for power cost minimization problem in gunbarrel natural gas networks.The demands between pipelines and compressor stations are uncertain with a budget parameter,since it is unlikely that all the uncertain demands reach the maximal deviation simultaneously.During solving the two-stage robust model we encounter a bilevel problem which is challenging to solve.We formulate it as a multi-dimensional dynamic programming problem and propose approximate dynamic programming methods to accelerate the calculation.Numerical results based on real network in China show that we obtain a speed gain of 7 times faster in average without compromising optimality compared with original dynamic programming algorithm.Numerical results also verify the advantage of robust model compared with deterministic model when facing uncertainties.These findings offer short-term operation methods for gunbarrel natural gas network management to handle with uncertainties.

Call for papers Journal of Control Theory and Applications Special issue on Approximate dynamic programming and reinforcement learning

Approximate dynamic programming (ADP) is a general and effective approach for solving optimal control and estimation problems by adapting to uncertain and nonconvex environments over time.

基于策略迭代ADP的碳纤维角联织机张力控制

针对碳纤维角联织机经纱张力控制问题,考虑开口等不确定因素对经纱张力的影响,建立了离散非线性送经系统张力控制模型,提出了策略迭代自适应动态规划(ADP),并对ADP中评价网络设计了自适应权值更新率;证明了策略迭代ADP在离散系统的收敛性,削减了非线性及不确定因素对经纱张力的影响,实现了对经纱张力的稳定控制,提高了系统鲁棒性。仿真结果表明:相比传统ADP,策略迭代ADP可以使经纱张力在2 s内快速无波动的到达稳定状态,使系统性能指标函数收敛更优。

Optimal Constrained Self-learning Battery Sequential Management in Microgrid Via Adaptive Dynamic Programming

This paper concerns a novel optimal self-learning battery sequential control scheme for smart home energy systems.The main idea is to use the adaptive dynamic programming(ADP) technique to obtain the optimal battery sequential control iteratively. First, the battery energy management system model is established, where the power efficiency of the battery is considered. Next, considering the power constraints of the battery, a new non-quadratic form performance index function is established, which guarantees that the value of the iterative control law cannot exceed the maximum charging/discharging power of the battery to extend the service life of the battery.Then, the convergence properties of the iterative ADP algorithm are analyzed, which guarantees that the iterative value function and the iterative control law both reach the optimums. Finally,simulation and comparison results are given to illustrate the performance of the presented method.

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems.Unlike existing optimal state feedback control,the control input of the optimal parallel control is introduced into the feedback system.However,due to the introduction of control input into the feedback system,the optimal state feedback control methods can not be applied directly.To address this problem,an augmented system and an augmented performance index function are proposed firstly.Thus,the general nonlinear system is transformed into an affine nonlinear system.The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically.It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function.Moreover,an adaptive dynamic programming(ADP)technique is utilized to implement the optimal parallel tracking control using a critic neural network(NN)to approximate the value function online.The stability analysis of the closed-loop system is performed using the Lyapunov theory,and the tracking error and NN weights errors are uniformly ultimately bounded(UUB).Also,the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals.Finally,the effectiveness of the developed optimal parallel control method is verified in two cases.

Optimal Control for a Class of Complex Singular System Based on Adaptive Dynamic Programming

This paper presents a new design approach to achieve decentralized optimal control of high-dimension complex singular systems with dynamic uncertainties. Based on robust adaptive dynamic programming(robust ADP) method, controllers for solving the singular systems optimal control problem are designed. The proposed algorithm can work well when the system model is not exactly known but the input and output data can be measured. The policy iteration of each controller only uses their own states and input information for learning,and do not need to know the whole system dynamics. Simulation results on the New England 10-machine 39-bus test system show the effectiveness of the designed controller.

Policy iteration optimal tracking control for chaotic systems by using an adaptive dynamic programming approach

A policy iteration algorithm of adaptive dynamic programming（ADP） is developed to solve the optimal tracking control for a class of discrete-time chaotic systems. By system transformations, the optimal tracking problem is transformed into an optimal regulation one. The policy iteration algorithm for discrete-time chaotic systems is first described. Then,the convergence and admissibility properties of the developed policy iteration algorithm are presented, which show that the transformed chaotic system can be stabilized under an arbitrary iterative control law and the iterative performance index function simultaneously converges to the optimum. By implementing the policy iteration algorithm via neural networks,the developed optimal tracking control scheme for chaotic systems is verified by a simulation.

新能源电力系统不确定优化调度方法研究现状及展望

风电场和光伏电站出力的不确定性给电力系统优化调度带来很大技术挑战。文中主要介绍了考虑新能源不确定性的电力系统优化调度方法的研究现状及后续研究方向展望。首先,重点论述了各种不确定优化调度(UOD)方法,包括随机优化方法、鲁棒优化方法、随机鲁棒优化结合方法和基于人工智能技术的方法。其中,随机优化方法包括场景法、机会约束规划法和近似动态规划法;鲁棒优化方法包括传统鲁棒优化法和分布鲁棒优化法;随机鲁棒优化结合方法包括采样鲁棒优化法和分布鲁棒机会约束规划法。然后,介绍了每一种方法的优化模型形式、模型的转化和求解原理及其优缺点。最后,对UOD的后续重点研究方向进行展望,包括兼顾多个目标的UOD问题及多目标不确定优化方法、输配系统UOD问题及分布式不确定优化方法、考虑稳定性约束的UOD问题及含常微分方程约束的不确定优化方法、考虑管道传输动态的综合能源系统UOD问题及含偏微分方程约束的不确定优化方法。

凸近似避障及采样区优化的智能车辆轨迹规划

针对结构化道路下作匀速运动的智能车辆避障轨迹规划问题,提出一种基于凸近似避障原理及采样区域优化的智能车辆轨迹规划方法。引入凸近似避障原理,得到轨迹可行域范围;将采样区域分为静态采样区、动态采样区两部分,并根据障碍物运动状态,另外划分动态、静态障碍物采样区;采用“动态规划(DP)+二次规划(QP)”思想求解轨迹:利用五次多项式对采样点依次连接,建立动态规划代价函数并筛选得到粗略轨迹;通过二次规划及约束条件的构造,对粗略轨迹进行平滑,最终得到最优轨迹。仿真结果表明:对于静态、低速、动态三种障碍物,该车能够有效地得到平滑轨迹并避开障碍物。

沿海区域综合能源生产单元随机优化调度

为实现沿海区域的海上风电场、海上采气平台和陆上热电联供燃气电厂等多种能源生产子单元的协同化运行,考虑可再生能源出力和氢负荷的随机波动,提出沿海区域综合能源生产单元(coastal integrated energy production units,CIEPU)随机优化调度模型。采用参数化代价函数近似(parametric cost function approximation,PCFA)的动态规划算法求解随机优化调度模型。通过一种基于梯度下降的求解方法--Adadelta法,获得策略函数的一阶信息,并计算梯度平方的指数衰减平均值,以更新策略函数的迭代步长;对随机优化调度模型进行策略参数逼近,从而得到近似最优的策略参数,并逐一时段求解出CIEPU的最优调度计划。最后,以某个CIEPU为例,分析计算结果表明,所提出方法获得的优化调度方案可以提高CIEPU运行的经济性并降低碳排放量,验证了所提方法的准确性和高效性。

基于ADP的复合燃料电池发电系统控制策略

在研究燃料电池/超级电容器复合电源结构基础上,引入了一种基于近似动态规划(Approxi-mate Dynamic Programming,ADP)的方法,用于燃料电池能量管理。详细介绍了ADP方法中执行依赖的启发式动态规划(Action-Depended Heuristic Dynamic Programming,ADHDP)的原理及构成。采用模糊神经网络产生控制信号,取代了传统由多层感知器所构成的执行网络,使得输入与输出之间的物理意义更为明确;采用增量输出的方式更适用于由于燃料利用率上下限所造成的强动态约束环境。针对复合发电系统的状态变量与控制信号,本文详尽阐述了评价网络与模糊神经网络的结构及其参数的调整方法。最后,仿真分析验证了控制策略的正确性以及良好的负载跟踪特性,并且在暂态过程中,燃料利用率仍在约束的范围内,保证了燃料电池的稳定运行与较高的发电效率。与传统的PI调节方式比较,本文采用的ADP方法具有响应快、超调小的特点。

加油站补货管理系统中在途库存的实时转运策略

加油站补货管理系统中的在途库存实时转运问题需要考虑转运时间以及未来不确定的需求,且需要实时响应转运需求。针对该难题,以最大化剩余决策时段内整体收益为目标,提出一种结合需求预测模型和随机需求分布以分段衡量整体收益的方法,解决了加油站补货管理系统中在途库存的实时转运难题。首先,将剩余决策时段划分为两个时段。其次,基于需求预测模型量化第1个时段内的需求,以衡量该时段的利润总和,并确定第2个时段初站点的库存水平;采用随机分布衡量需求的不确定性,基于近似动态规划算法求解库存水平的值函数,以衡量第2个时段的利润总和;设计拟被转运走部分在途库存的站点的选择规则,缩小决策空间以提高策略的响应效率。最后,对比实验验证了本文方法产生的转运策略在不同站点规模和初始库存范围下的有效性,并得出管理启示。本研究可为加油站补货管理系统中在途库存的实时转运提供决策支持,对类似需同时考虑转运时间和未来需求的实时转运问题具有启发意义。