Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems.Unlike existing optimal state feedback control,the control input of the optimal parallel control is introduced into the feedback system.However,due to the introduction of control input into the feedback system,the optimal state feedback control methods can not be applied directly.To address this problem,an augmented system and an augmented performance index function are proposed firstly.Thus,the general nonlinear system is transformed into an affine nonlinear system.The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically.It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function.Moreover,an adaptive dynamic programming(ADP)technique is utilized to implement the optimal parallel tracking control using a critic neural network(NN)to approximate the value function online.The stability analysis of the closed-loop system is performed using the Lyapunov theory,and the tracking error and NN weights errors are uniformly ultimately bounded(UUB).Also,the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals.Finally,the effectiveness of the developed optimal parallel control method is verified in two cases.

Optimal Constrained Self-learning Battery Sequential Management in Microgrid Via Adaptive Dynamic Programming

This paper concerns a novel optimal self-learning battery sequential control scheme for smart home energy systems. The main idea is to use the adaptive dynamic programming U+0028 ADP U+0029 technique to obtain the optimal battery sequential control iteratively. First, the battery energy management system model is established, where the power efficiency of the battery is considered. Next, considering the power constraints of the battery, a new non-quadratic form performance index function is established, which guarantees that the value of the iterative control law cannot exceed the maximum charging/discharging power of the battery to extend the service life of the battery. Then, the convergence properties of the iterative ADP algorithm are analyzed, which guarantees that the iterative value function and the iterative control law both reach the optimums. Finally, simulation and comparison results are given to illustrate the performance of the presented method. © 2017 Chinese Association of Automation.

Optimal Control for a Class of Complex Singular System Based on Adaptive Dynamic Programming

This paper presents a new design approach to achieve decentralized optimal control of high-dimension complex singular systems with dynamic uncertainties. Based on robust adaptive dynamic programming(robust ADP) method, controllers for solving the singular systems optimal control problem are designed. The proposed algorithm can work well when the system model is not exactly known but the input and output data can be measured. The policy iteration of each controller only uses their own states and input information for learning,and do not need to know the whole system dynamics. Simulation results on the New England 10-machine 39-bus test system show the effectiveness of the designed controller.

Value Iteration-Based Cooperative Adaptive Optimal Control for Multi-Player Differential Games With Incomplete Information

This paper presents a novel cooperative value iteration(VI)-based adaptive dynamic programming method for multi-player differential game models with a convergence proof.The players are divided into two groups in the learning process and adapt their policies sequentially.Our method removes the dependence of admissible initial policies,which is one of the main drawbacks of the PI-based frameworks.Furthermore,this algorithm enables the players to adapt their control policies without full knowledge of others’ system parameters or control laws.The efficacy of our method is illustrated by three examples.

基于策略迭代ADP的碳纤维角联织机张力控制

针对碳纤维角联织机经纱张力控制问题,考虑开口等不确定因素对经纱张力的影响,建立了离散非线性送经系统张力控制模型,提出了策略迭代自适应动态规划(ADP),并对ADP中评价网络设计了自适应权值更新率;证明了策略迭代ADP在离散系统的收敛性,削减了非线性及不确定因素对经纱张力的影响,实现了对经纱张力的稳定控制,提高了系统鲁棒性。仿真结果表明:相比传统ADP,策略迭代ADP可以使经纱张力在2 s内快速无波动的到达稳定状态,使系统性能指标函数收敛更优。

Discounted Iterative Adaptive Critic Designs With Novel Stability Analysis for Tracking Control

The core task of tracking control is to make the controlled plant track a desired trajectory.The traditional performance index used in previous studies cannot eliminate completely the tracking error as the number of time steps increases.In this paper,a new cost function is introduced to develop the value-iteration-based adaptive critic framework to solve the tracking control problem.Unlike the regulator problem,the iterative value function of tracking control problem cannot be regarded as a Lyapunov function.A novel stability analysis method is developed to guarantee that the tracking error converges to zero.The discounted iterative scheme under the new cost function for the special case of linear systems is elaborated.Finally,the tracking performance of the present scheme is demonstrated by numerical results and compared with those of the traditional approaches.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems,in this paper,a new iterative adaptive dynamic programming algorithm,which is the discrete-time time-varying policy iteration(DTTV)algorithm,is developed.The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance.The admissibility of the iterative control law is analyzed.The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution.To implement the algorithm,neural networks are employed and a new implementation structure is established,which avoids solving the generalized Bellman equation in each iteration.Finally,the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm,where the mass and pendulum bar length are permitted to be time-varying parameters.The effectiveness of the developed method is illustrated by numerical results and comparisons.

A novel stable value iteration-based approximate dynamic programming algorithm for discrete-time nonlinear systems

The convergence and stability of a value-iteration-based adaptive dynamic programming （ADP） algorithm are con- sidered for discrete-time nonlinear systems accompanied by a discounted quadric performance index. More importantly than sufficing to achieve a good approximate structure, the iterative feedback control law must guarantee the closed-loop stability. Specifically, it is firstly proved that the iterative value function sequence will precisely converge to the optimum. Secondly, the necessary and sufficient condition of the optimal value function serving as a Lyapunov function is investi- gated. We prove that for the case of infinite horizon, there exists a finite horizon length of which the iterative feedback control law will provide stability, and this increases the practicability of the proposed value iteration algorithm. Neural networks （NNs） are employed to approximate the value functions and the optimal feedback control laws, and the approach allows the implementation of the algorithm without knowing the internal dynamics of the system. Finally, a simulation example is employed to demonstrate the effectiveness of the developed optimal control method.

近似动态规划在电力系统优化运行中的应用综述

高比例可再生能源的接入和日益扩大的系统规模给电力系统优化运行带来了巨大挑战。近似动态规划(ADP)能够有效处理维数灾问题,在求解复杂系统动态优化问题上具备强大优势,近年来成为运筹学领域的一大研究热点。文中对ADP及其在电力优化运行中的应用进行了综述。首先,阐述了ADP的基本思想、主要特点及其近似策略。其次,介绍了不同类型的ADP在电力系统优化运行中的应用现状。最后,讨论了当前ADP在理论和应用中的不足,并对其未来发展方向作了进一步的展望。

基于IDP的重型商用车自适应距离域预见性巡航控制策略

为降低重型商用车燃油消耗、减少运输成本,本文协调“人-车-路”交互体系,将车辆与智能网联环境下的多维度信息进行融合,提出了一种基于迭代动态规划(iterative dynamic programming,IDP)的自适应距离域预见性巡航控制策略(adaptive range predictive cruise control strategy,ARPCC)。首先结合车辆状态与前方环境多维度信息,基于车辆纵向动力学建立自适应距离域模型对路网重构,简化网格数量并利用IDP求取全局最优速度序列。其次,在全局最优速度序列的基础上,求取自适应距离域内的分段最优速度序列,实现车辆控制状态的快速求解。最后,利用Matlab/Simulink进行验证。结果表明,通过多次迭代缩小网格,该算法有效提高了计算效率和车辆燃油经济性。

基于多人零和博弈的模块化机器人系统近似最优控制

提出一种基于多人零和博弈的模块化机器人(Modular Robot Manipulators, MRMs)系统近似最优控制方法。建立了具有交联耦合(Interconnected Dynamic Couplings, IDC)的模块化机器人系统动力学模型。将机器人系统的控制律和IDC效应作为零和博弈的参与者,MRM系统的最优跟踪控制问题转化为多人零和博弈问题。根据自适应动态规划(Adaptive Dynamic Programming, ADP)算法,通过建立评判神经网络求解哈密顿-雅克比-埃塞克斯(Hamilton-Jacobi-Issacs, HJI)方程,推导出最优控制律。基于李雅普诺夫定理,证明了闭环机器人系统是渐近稳定的,最后通过实验验证了所提控制方法的有效性。

永磁同步电动机速度伺服系统最优输出反馈控制器设计

针对永磁同步电动机(Permanent magnet synchronous motor,PMSM)模型参数未知以及电枢电流和负载转矩无法直接测量的问题,设计一种基于自适应动态规划(Adaptive dynamic programming,ADP)的输出反馈控制方案,实现PMSM最优速度跟踪控制.首先,根据PMSM内部特性确定其数学模型的结构,构建与原始系统相对应的辅助系统,引入新的线性二次指标来实现速度最优跟踪调节.其次,设计一种嵌入式观测器,该观测器能够在系统模型未知情况下用可测量数据重构系统全部状态.此外,提出一种离线策略的ADP方法逼近最优控制增益的解.最后,仿真结果验证所提控制方案在模型参数未知以及电枢电流和负载转矩不可测量的情况下,实现了精确的速度跟踪性能和良好的瞬态响应,同时降低了电压的冲击.

基于ADP的一类时滞离散系统跟踪控制

时滞现象是自然界中广泛存在的一种物理现象,时滞的存在使得被控量不能及时反映系统的变化,从而使控制系统的稳定性变差,给时滞系统控制器的设计带来很大困难。针对一类状态和控制输入均含有时滞的离散仿射系统的跟踪控制进行研究,采用自适应动态规划迭代算法求解时滞系统的跟踪控制,在自适应动态规划的基础上,建立系统性能指标函数,通过系统变换将跟踪问题转化成为最优调节问题,并采用自适应动态规划迭代算法对性能指标函数进行迭代求解,得到最优控制策略。并给出了一个仿真算例,结果证明了所提出的跟踪控制方案是有效的。

基于ELM的水泥立磨生料细度ADP控制

水泥生产中的立磨粉磨过程具有非线性、强耦合、大滞后等特点,对其进行精确的建模和实现生料细度的控制比较困难。提出一种基于极限学习机(ELM,extreme learning machine)的自适应动态规划(ADP,adaptive dynamic programming)优化控制算法。采用极限学习机建立立磨生料粉磨过程的生料细度预测模型,将其作为ADP算法中的模型网络,并以在线序列极限学习机实现ADP的执行网络和评价网络。结果表明:在仿真意义上,所提算法能够对生料细度进行有效地控制,对立磨稳定生产,降低该生产过程的能耗具有一定理论指导意义。

基于航迹消除与策略迭代的无人机集群区域目标搜索方法

无人机集群区域搜索在军事领域以及民用领域的搜救、巡逻、监测、环境勘测等方面有着广泛的应用,但如何保证不同场景下无人机集群搜索方法的效率问题依然是个难题。为了更好地解决搜索目标先验信息已知的无障碍区域内多无人机集群搜索航迹规划问题,提高无人机集群搜索效率,本文根据目标区域热度以及传感器探测概率等先验信息,提出了一种基于无人机航迹消除策略的概率计算方法,并在此基础上结合策略迭代算法动态规划无人机航迹,找到单个无人机航迹覆盖率最优策略;进而通过适当组合顺序实现无人机集群区域目标搜索整体覆盖率最优;最后,通过仿真计算验证了算法的有效性。

多智能体系统事件触发固定时间最优一致性

针对多智能体系统(multi-agent systems,MASs)固定时间最优领导-跟随一致性问题,基于性能优化目标,设计了一种基于事件触发机制的最优控制策略,兼顾固定时间最优一致性控制目标和有限的系统通信计算资源。为了近似求解Hamilton-Jacobi-Bellman(HJB)方程获得最优值函数的表达式,提出一种仅包含Critic神经网络结构的自适应动态规划(adaptive dynamic programming,ADP)在线学习算法,结合梯度下降法和经验重放方法,利用历史记录数据和当前数据更新神经网络权重向量近似最优值函数及其梯度。采用基于无人地面车辆的无人集群系统验证了该方法的可行性。