Policy iteration optimal tracking control for chaotic systems by using an adaptive dynamic programming approach

A policy iteration algorithm of adaptive dynamic programming(ADP) is developed to solve the optimal tracking control for a class of discrete-time chaotic systems. By system transformations, the optimal tracking problem is transformed into an optimal regulation one. The policy iteration algorithm for discrete-time chaotic systems is first described. Then,the convergence and admissibility properties of the developed policy iteration algorithm are presented, which show that the transformed chaotic system can be stabilized under an arbitrary iterative control law and the iterative performance index function simultaneously converges to the optimum. By implementing the policy iteration algorithm via neural networks,the developed optimal tracking control scheme for chaotic systems is verified by a simulation.

Optimal Constrained Self-learning Battery Sequential Management in Microgrid Via Adaptive Dynamic Programming

This paper concerns a novel optimal self-learning battery sequential control scheme for smart home energy systems.The main idea is to use the adaptive dynamic programming(ADP) technique to obtain the optimal battery sequential control iteratively. First, the battery energy management system model is established, where the power efficiency of the battery is considered. Next, considering the power constraints of the battery, a new non-quadratic form performance index function is established, which guarantees that the value of the iterative control law cannot exceed the maximum charging/discharging power of the battery to extend the service life of the battery.Then, the convergence properties of the iterative ADP algorithm are analyzed, which guarantees that the iterative value function and the iterative control law both reach the optimums. Finally,simulation and comparison results are given to illustrate the performance of the presented method.

Adaptive dynamic programming for online solution of a zero-sum differential game

This paper will present an approximate/adaptive dynamic programming(ADP) algorithm,that uses the idea of integral reinforcement learning(IRL),to determine online the Nash equilibrium solution for the two-player zerosum differential game with linear dynamics and infinite horizon quadratic cost.The algorithm is built around an iterative method that has been developed in the control engineering community for solving the continuous-time game algebraic Riccati equation(CT-GARE),which underlies the game problem.We here show how the ADP techniques will enhance the capabilities of the offline method allowing an online solution without the requirement of complete knowledge of the system dynamics.The feasibility of the ADP scheme is demonstrated in simulation for a power system control application.The adaptation goal is the best control policy that will face in an optimal manner the highest load disturbance.

State of the Art of Adaptive Dynamic Programming and Reinforcement Learning

This article introduces the state-of-the-art development of adaptive dynamic programming and reinforcement learning(ADPRL).First,algorithms in reinforcement learning(RL)are introduced and their roots in dynamic programming are illustrated.Adaptive dynamic programming(ADP)is then introduced following a brief discussion of dynamic programming.Researchers in ADP and RL have enjoyed the fast developments of the past decade from algorithms,to convergence and optimality analyses,and to stability results.Several key steps in the recent theoretical developments of ADPRL are mentioned with some future perspectives.In particular,convergence and optimality results of value iteration and policy iteration are reviewed,followed by an introduction to the most recent results on stability analysis of value iteration algorithms.

Approximation-error-ADP-based optimal tracking control for chaotic systems with convergence proof

In this paper, an optimal tracking control scheme is proposed for a class of discrete-time chaotic systems using the approximation-error-based adaptive dynamic programming (ADP) algorithm. Via the system transformation, the optimal tracking problem is transformed into an optimal regulation problem, and then the novel optimal tracking control method is proposed. It is shown that for the iterative ADP algorithm with finite approximation error, the iterative performance index functions can converge to a finite neighborhood of the greatest lower bound of all performance index functions under some convergence conditions. Two examples are given to demonstrate the validity of the proposed optimal tracking control scheme for chaotic systems.

Chaotic system optimal tracking using data-based synchronous method with unknown dynamics and disturbances

We develop an optimal tracking control method for chaotic system with unknown dynamics and disturbances. The method allows the optimal cost function and the corresponding tracking control to update synchronously. According to the tracking error and the reference dynamics, the augmented system is constructed. Then the optimal tracking control problem is defined. The policy iteration(PI) is introduced to solve the min-max optimization problem. The off-policy adaptive dynamic programming(ADP) algorithm is then proposed to find the solution of the tracking Hamilton–Jacobi–Isaacs(HJI) equation online only using measured data and without any knowledge about the system dynamics. Critic neural network(CNN), action neural network(ANN), and disturbance neural network(DNN) are used to approximate the cost function, control, and disturbance. The weights of these networks compose the augmented weight matrix, and the uniformly ultimately bounded(UUB) of which is proven. The convergence of the tracking error system is also proven. Two examples are given to show the effectiveness of the proposed synchronous solution method for the chaotic system tracking problem.

Approximate dynamic programming solutions with a single network adaptive critic for a class of nonlinear systems

Approximate dynamic programming(ADP) formulation implemented with an adaptive critic(AC)-based neural network(NN) structure has evolved as a powerful technique for solving the Hamilton-Jacobi-Bellman(HJB) equations.As interest in ADP and the AC solutions are escalating with time,there is a dire need to consider possible enabling factors for their implementations.A typical AC structure consists of two interacting NNs,which is computationally expensive.In this paper,a new architecture,called the 'cost-function-based single network adaptive critic(J-SNAC)' is presented,which eliminates one of the networks in a typical AC structure.This approach is applicable to a wide class of nonlinear systems in engineering.In order to demonstrate the benefits and the control synthesis with the J-SNAC,two problems have been solved with the AC and the J-SNAC approaches.Results are presented,which show savings of about 50% of the computational costs by J-SNAC while having the same accuracy levels of the dual network structure in solving for optimal control.Furthermore,convergence of the J-SNAC iterations,which reduces to a least-squares problem,is discussed;for linear systems,the iterative process is shown to reduce to solving the familiar algebraic Ricatti equation.

Off-policy integral reinforcement learning optimal tracking control for continuous-time chaotic systems

This paper estimates an off-policy integral reinforcement learning(IRL) algorithm to obtain the optimal tracking control of unknown chaotic systems. Off-policy IRL can learn the solution of the HJB equation from the system data generated by an arbitrary control. Moreover, off-policy IRL can be regarded as a direct learning method, which avoids the identification of system dynamics. In this paper, the performance index function is first given based on the system tracking error and control error. For solving the Hamilton–Jacobi–Bellman(HJB) equation, an off-policy IRL algorithm is proposed.It is proven that the iterative control makes the tracking error system asymptotically stable, and the iterative performance index function is convergent. Simulation study demonstrates the effectiveness of the developed tracking control method.

Discounted Iterative Adaptive Critic Designs With Novel Stability Analysis for Tracking Control

The core task of tracking control is to make the controlled plant track a desired trajectory.The traditional performance index used in previous studies cannot eliminate completely the tracking error as the number of time steps increases.In this paper,a new cost function is introduced to develop the value-iteration-based adaptive critic framework to solve the tracking control problem.Unlike the regulator problem,the iterative value function of tracking control problem cannot be regarded as a Lyapunov function.A novel stability analysis method is developed to guarantee that the tracking error converges to zero.The discounted iterative scheme under the new cost function for the special case of linear systems is elaborated.Finally,the tracking performance of the present scheme is demonstrated by numerical results and compared with those of the traditional approaches.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems,in this paper,a new iterative adaptive dynamic programming algorithm,which is the discrete-time time-varying policy iteration(DTTV)algorithm,is developed.The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance.The admissibility of the iterative control law is analyzed.The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution.To implement the algorithm,neural networks are employed and a new implementation structure is established,which avoids solving the generalized Bellman equation in each iteration.Finally,the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm,where the mass and pendulum bar length are permitted to be time-varying parameters.The effectiveness of the developed method is illustrated by numerical results and comparisons.

基于数据的自学习优化控制:研究进展与展望

自适应动态规划(Adaptive dynamic programming,ADP)方法可以解决传统动态规划中的"维数灾"问题,已经成为控制理论和计算智能领域最新的研究热点.ADP方法采用函数近似结构来估计系统性能指标函数,然后依据最优性原理来获得近优的控制策略.ADP是一种具有学习和优化能力的智能控制方法,在求解复杂非线性系统的最优控制问题中具有极大的潜力.本文对ADP的理论研究、算法实现、相关应用等方面进行了全面的梳理,涵盖了最新的研究进展,并对ADP的未来发展趋势进行了分析和展望.

微电网多目标随机动态优化调度算法

针对含风光发电和储能电池的微电网多目标随机动态优化调度问题,建立以微电源总运行费用和系统总网损为目标函数,同时以多个蓄电池剩余电量的和作为资源存储量的微电网多目标随机型存储模型。模型中采用交流潮流模型准确描述配电线路的传输功率安全约束,并考虑了各种分布式电源的电压无功特性。结合自适应加权和法(AWS)和近似动态规划法(ADP)求解多目标随机动态优化调度问题,先采用AWS法将多目标随机动态优化模型转化为一系列单目标随机动态优化模型,再采用ADP的近似值函数迭代算法实现对单目标随机动态优化模型的逐时段递推解耦求解,并通过对AWS法中分割段新增Pareto点对应权值的调整以得到均匀分布的Pareto前沿。通过某一实际微电网的算例仿真,证明了所提出模型与算法的正确性和有效性。

自适应单指数平滑法在短期交通流预测中的应用

短时交通流预测是实现交通规划和管理的关键技术之一.指数平滑法因其计算过程简单,需要观测数据较少等优点,在短时交通流预测中获得了广泛的应用,但其平滑系数缺乏有效的选取方法.本文提出了一种自适应单指数平滑法,通过近似动态规划方法的引入,结合实际交通流数据对指数平滑系数进行优化,使其随预测过程自动更新,从而保证了预测的实时性、准确性.严格的理论推导证明了这种预测方法的收敛性,仿真结果验证了算法的有效性.

一种新型附加学习控制器及电力系统应用实例

维纳的《控制论》和钱学森的《工程控制论》共同奠定了经典控制理论的基础.在此基础之上,现代控制理论在性能优化和处理不确定性等方面对经典控制理论作了进一步的发展.本文提出一种融合经典控制与现代控制的控制方法,将属于现代控制的近似动态规划学习控制器以并联的方式附加到经典控制器上,从而形成一类新型附加学习控制器.该控制器采用基于策略迭代近似动态规划的训练算法和基于最小二乘的代价函数逼近算法,从而具有高策略搜索效率和高样本利用效率,其中动作依赖代价函数的引入使得在线学习不依赖系统模型.总之,所提附加学习控制器一方面融合了已有经典控制器的先验知识,另一方面为已有经典控制器提供了可设计的目标函数和自趋优机制.论文进一步从理论上严格证明了所提附加学习控制方法的稳定性和收敛性.针对双馈风电场暂态无功控制问题的仿真研究验证了所提附加学习控制器的正确性和方法的有效性.

基于功能度量法的概率优化设计的收敛控制

概率结构优化设计(PSDO)中概率约束的评定可以采用最近提出的、被认为更高效、稳定的功能度量法(PMA).改进均值(AMV)迭代格式经常在PMA中使用,但它对一些非线性功能函数或非正态随机变量,搜索最小功能目标点时可能陷入周期振荡或混沌解,从而使PSDO的两层次算法或序列近似规划算法优化计算失败.利用混沌反馈控制的稳定转换法对功能度量法的AMV迭代格式实施了收敛控制,使嵌入周期和混沌轨道的不稳定不动点稳定化,获得稳定收敛解,从而使概率约束的评定能正常进行;再由两层次算法或序列近似规划算法进行结构优化设计.算例结果表明了稳定转换法实施收敛控制的有效性,以及序列近似规划算法相对高效的优点.

Data-Based Optimal Tracking of Autonomous Nonlinear Switching Systems

In this paper,a data-based scheme is proposed to solve the optimal tracking problem of autonomous nonlinear switching systems.The system state is forced to track the reference signal by minimizing the performance function.First,the problem is transformed to solve the corresponding Bellman optimality equation in terms of the Q-function(also named as action value function).Then,an iterative algorithm based on adaptive dynamic programming(ADP)is developed to find the optimal solution which is totally based on sampled data.The linear-in-parameter(LIP)neural network is taken as the value function approximator.Considering the presence of approximation error at each iteration step,the generated approximated value function sequence is proved to be boundedness around the exact optimal solution under some verifiable assumptions.Moreover,the effect that the learning process will be terminated after a finite number of iterations is investigated in this paper.A sufficient condition for asymptotically stability of the tracking error is derived.Finally,the effectiveness of the algorithm is demonstrated with three simulation examples.

基于非线性多输入多输出近似动态规划的发动机缸平衡智能调节算法

为解决发动机缸平衡控制问题,将标准近似动态规划(approximate dynamic programming,ADP)扩展为多输入多输出形式,给出其控制算法,并证明其收敛性。仿真结果显示,该方法能在一定范围内智能地调节各缸喷油量,用于补偿由多种不确定因素导致的各缸转速差异,从而自适应地提高缸平衡效果。该方法只需基于实时转速,不必检测和区分各缸间的转速差异,具有非线性系统的智能优化特点。该方法能直接处理各缸间的非线性多输入多输出耦合关系。

一种考虑货位共享效应的Fishbone仓储布局优化方法

以Fishbone仓储布局为基础,针对3种经典存储策略,考虑货位共享效应,以单程平均货物拣选距离最短为目标,建立Fishbone布局仓库设计优化模型,以探讨将货位共享效应考虑在内时,不同存储策略对仓储布局的影响;采用分次逼近策略和动态规划算法确定货物的优化分类及类别边界;设计基于自适应的遗传算法获得最优存储分类下的仓储布局参数,并结合案例实验数据进行仿真分析。结果表明,基于分类存储策略的仓储布局表现优异,所需货位数量少且货物拣选距离短;另外,仓储存储物品的需求差异越大,考虑货位共享效应的优势就越显著,最高可减少37.1%的货物拣选距离。

基于广义模糊双曲模型的自适应动态规划最优控制设计

为连续非线性系统提出了一种有效的最优控制设计方法.广义模糊双曲模型(Generalized fuzzy hyperbolic model,GFHM)首次作为逼近器用来估计HJB(Hamilton-Jacobi-Bellman)方程的解(值函数,即它是状态与代价函数之间的映射),然后,利用该近似解获得最优控制.本文方法只需要一个GFHM估计值函数.首先,阐述了对于连线非线性系统最优控制的设计过程;然后,证明了逼近误差是一致最终有界的(Uniformly ultimately bounded,UUB);最后,一个数值例子验证了本文方法的有效性.另一个例子通过与神经网络自适应动态规划的方法作比较,演示了本文方法的优点.