Science Letters：On stochastic optimal control of partially observable nonlinear quasi Hamiltonian systems

A stochastic optimal control strategy for partially observable nonlinear quasi Hamiltonian systems is proposed. The optimal control forces consist of two parts. The first part is determined by the conditions under which the stochastic optimal control problem of a partially observable nonlinear system is converted into that of a completely observable linear system. The second part is determined by solving the dynamical programming equation derived by applying the stochastic averaging method and stochastic dynamical programming principle to the completely observable linear control system. The response of the optimally controlled quasi Hamiltonian system is predicted by solving the averaged Fokker-Planck-Kolmogorov equation associated with the optimally controlled completely observable linear system and solving the Riccati equation for the estimated error of system states. An example is given to illustrate the procedure and effectiveness of the proposed control strategy.

A NEW STOCHASTIC OPTIMAL CONTROL STRATEGY FOR HYSTERETIC MR DAMPERS

A new stochastic optimal control strategy for randomly excited quasi-integrable Hamiltonian systems using magneto-rheological (MR) dampers is proposed. The dynamic be- havior of an MR damper is characterized by the Bouc-Wen hysteretic model. The control force produced by the MR damper is separated into a passive part incorporated in the uncontrolled system and a semi-active part to be determined. The system combining the Bouc-Wen hysteretic force is converted into an equivalent non-hysteretic nonlinear stochastic control system. Then It?o stochastic di?erential equations are derived from the equivalent system by using the stochastic averaging method. A dynamical programming equation for the controlled di?usion processes is established based on the stochastic dynamical programming principle. The non-clipping nonlin- ear optimal control law is obtained for a certain performance index by minimizing the dynamical programming equation. Finally, an example is given to illustrate the application and e?ectiveness of the proposed control strategy.

Numerical Solution of a Class of Nonlinear Optimal Control Problems Using Linearization and Discretization

In this paper, a new approach using linear combination property of intervals and discretization is proposed to solve a class of nonlinear optimal control problems, containing a nonlinear system and linear functional, in three phases. In the first phase, using linear combination property of intervals, changes nonlinear system to an equivalent linear system, in the second phase, using discretization method, the attained problem is converted to a linear programming problem, and in the third phase, the latter problem will be solved by linear programming methods. In addition, efficiency of our approach is confirmed by some numerical examples.

Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems.Unlike existing optimal state feedback control,the control input of the optimal parallel control is introduced into the feedback system.However,due to the introduction of control input into the feedback system,the optimal state feedback control methods can not be applied directly.To address this problem,an augmented system and an augmented performance index function are proposed firstly.Thus,the general nonlinear system is transformed into an affine nonlinear system.The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically.It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function.Moreover,an adaptive dynamic programming(ADP)technique is utilized to implement the optimal parallel tracking control using a critic neural network(NN)to approximate the value function online.The stability analysis of the closed-loop system is performed using the Lyapunov theory,and the tracking error and NN weights errors are uniformly ultimately bounded(UUB).Also,the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals.Finally,the effectiveness of the developed optimal parallel control method is verified in two cases.

A Novel Distributed Optimal Adaptive Control Algorithm for Nonlinear Multi-Agent Differential Graphical Games

In this paper, an online optimal distributed learning algorithm is proposed to solve leader-synchronization problem of nonlinear multi-agent differential graphical games. Each player approximates its optimal control policy using a single-network approximate dynamic programming(ADP) where only one critic neural network(NN) is employed instead of typical actorcritic structure composed of two NNs. The proposed distributed weight tuning laws for critic NNs guarantee stability in the sense of uniform ultimate boundedness(UUB) and convergence of control policies to the Nash equilibrium. In this paper, by introducing novel distributed local operators in weight tuning laws, there is no more requirement for initial stabilizing control policies. Furthermore, the overall closed-loop system stability is guaranteed by Lyapunov stability analysis. Finally, Simulation results show the effectiveness of the proposed algorithm.

Optimal control strategies for stochastically excited quasi partially integrable Hamiltonian systems

In this paper two different control strategies designed to alleviate the response of quasi partially integrable Hamiltonian systems subjected to stochastic excitation are proposed. First, by using the stochastic averaging method for quasi partially integrable Hamiltonian systems, an n-DOF controlled quasi partially integrable Hamiltonian system with stochastic excitation is converted into a set of partially averaged It^↑o stochastic differential equations. Then, the dynamical programming equation associated with the partially averaged It^↑o equations is formulated by applying the stochastic dynamical programming principle. In the first control strategy, the optimal control law is derived from the dynamical programming equation and the control constraints without solving the dynamical programming equation. In the second control strategy, the optimal control law is obtained by solving the dynamical programming equation. Finally, both the responses of controlled and uncontrolled systems are predicted through solving the Fokker-Plank-Kolmogorov equation associated with fully averaged It^↑o equations. An example is worked out to illustrate the application and effectiveness of the two proposed control strategies.

Dynamic Programming to Identification Problems

An identification problem is considered as inaccurate measurements of dynamics on a time interval are given. The model has the form of ordinary differential equations which are linear with respect to unknown parameters. A new approach is presented to solve the identification problem in the framework of the optimal control theory. A numerical algorithm based on the dynamic programming method is suggested to identify the unknown parameters. Results of simulations are exposed.

Approximate dynamic programming solutions with a single network adaptive critic for a class of nonlinear systems

Approximate dynamic programming(ADP) formulation implemented with an adaptive critic(AC)-based neural network(NN) structure has evolved as a powerful technique for solving the Hamilton-Jacobi-Bellman(HJB) equations.As interest in ADP and the AC solutions are escalating with time,there is a dire need to consider possible enabling factors for their implementations.A typical AC structure consists of two interacting NNs,which is computationally expensive.In this paper,a new architecture,called the ’cost-function-based single network adaptive critic(J-SNAC)’ is presented,which eliminates one of the networks in a typical AC structure.This approach is applicable to a wide class of nonlinear systems in engineering.In order to demonstrate the benefits and the control synthesis with the J-SNAC,two problems have been solved with the AC and the J-SNAC approaches.Results are presented,which show savings of about 50% of the computational costs by J-SNAC while having the same accuracy levels of the dual network structure in solving for optimal control.Furthermore,convergence of the J-SNAC iterations,which reduces to a least-squares problem,is discussed;for linear systems,the iterative process is shown to reduce to solving the familiar algebraic Ricatti equation.

基于迭代神经动态规划的数据驱动非线性近似最优调节

利用数据驱动控制思想,建立一种设计离散时间非线性系统近似最优调节器的迭代神经动态规划方法.提出针对离散时间一般非线性系统的迭代自适应动态规划算法并且证明其收敛性与最优性.通过构建三种神经网络,给出全局二次启发式动态规划技术及其详细的实现过程,其中执行网络是在神经动态规划的框架下进行训练.这种新颖的结构可以近似代价函数及其导函数,同时在不依赖系统动态的情况下自适应地学习近似最优控制律.值得注意的是,这在降低对于控制矩阵或者其神经网络表示的要求方面,明显地改进了迭代自适应动态规划算法的现有结果,能够促进复杂非线性系统基于数据的优化与控制设计的发展.通过两个仿真实验,验证本文提出的数据驱动最优调节方法的有效性.

带有储能设备的智能电网电能迭代自适应动态规划最优控制

智能电网是新一代电网建设的目标,也是国际电力工业界的共同选择.本文研究在储能设备接入电网情况下,建立一套基于自适应动态规划(Adaptive dynamic programming,ADP)的智能电网电能自适应优化控制的理论与方法,实现电网发电端以及用户端的智能交互,开辟对智能电网供需优化匹配与调控方法的新途径.论文首先给出动态规划的最优性原理以及带有储能设备智能电网的运行方式并提出优化目标;然后,设计新型迭代自适应动态规划方法实现对储能设备的最优控制,并证明自适应动态规划方法的收敛性,在理论上保证了对智能电网电能的优化;最后,给出仿真例子显示出所提出控制方法的有效性.

模型预测暂态电压稳定紧急控制的简化空间算法

采用模型预测进行暂态电压稳定紧急控制,计算效率是制约其在线应用的瓶颈。建立暂态电压稳定紧急控制的滚动优化模型。采用拉道(Radau)排列法将这个动态优化模型转换为非线性规划模型,转换之后的非线性规划模型具有高维但自由度相对低的特点。为提高计算效率,利用其自由度低的特征,采用简化空间序列二次规划算法进行求解。并对简化空间序列二次规划算法的几个关键问题进行讨论。以IEEE 3机9节点和IEEE 10机39节点系统作为测试系统,通过与全空间序列二次规划算法、标准内点法和基于有限内存存储技术的内点法的比较,验证所提算法的有效性。

基于数据的智能电网电能自适应优化调控

智能电网是新一代电网建设的目标,也是国际电力工业界的共同选择。首次提出基于数据的自适应动态规划的智能电网电能自适应优化控制方法。无需智能电网的模型,采用智能电网数据获得智能电网的最优控制策略,有效地克服了智能电网系统模型难以建立的困难。首先,给出了智能电网的具体工作原理,列出了智能电网储能设备,风能、太阳能等清洁能源的运行原理。其次给出了基于数据的智能电网电能自适应优化调控方案。然后讨论了基于神经网络的自适应优化具体实现方案。最后给出仿真实例证明方法的有效性。

具有条件马尔科夫结构的离散随机系统最优控制

基于Bellman随机非线性动态规划法,提出了具有条件马尔科夫跳变结构的离散随机系统的最优控制方法,应用随机变结构系统的性质对最优控制算法进行了简化处理,并将后验概率密度函数用条件高斯函数来逼近,针对一类具有条件马尔科夫跳变结构的线性离散随机系统,给出了其逼近最优控制算法.

拟哈密顿系统非线性随机最优控制

主要介绍近十几年来拟哈密顿系统非线性随机最优控制理论方法及其应用的研究成果,包括基于拟哈密顿系统随机平均法与随机动态规划原理的非线性随机最优控制基本策略,即响应极小化控制、随机稳定化、首次穿越损坏最小化控制、以概率密度为目标的控制,为将它们应用于工程实际而作的部分可观测系统最优控制、有界控制、时滞控制、半主动控制、极小极大控制的进一步研究,以及综合考虑这些实际问题的非线性随机最优控制的综合策略,非线性随机最优控制在滞迟系统、分数维系统等中的若干应用,介绍与这些研究有关的背景,并指出今后有待进一步研究的问题.

执行器故障不确定非线性系统最优自适应输出跟踪控制

本文针对一类具有执行器故障的多输入多输出(Multi-input multi-output, MIMO)不确定连续仿射非线性系统,提出了一种最优自适应输出跟踪控制方案.设计了保证系统稳定性的不确定项估计神经网络权值调整算法,仅采用评价网络即可同时获得无限时域代价函数和满足哈密顿–雅可比–贝尔曼(Hamilton-Jacobi-Bellman, HJB)方程的最优控制输入.考虑执行器卡死和部分失效故障,设计最优自适应补偿控制律,所设计的控制律可以实现对参考输出的一致最终有界跟踪.飞行器控制仿真和对比验证表明了本文方法的有效性和优越性.

带ε误差限的近似最优控制

近似动态规划方法求解非线性系统最优控制,需要迭代无限步才能得到最优控制律.本文提出了一种ε–近似最优控制算法,选择ε误差限,通过自适应迭代不断逼近哈密顿–雅可比–贝尔曼(HJB)方程的解,应用神经网络实现在有限步迭代后得到带ε误差限的近似最优控制律.计算机仿真结果表明了该算法的有效性.

非线性零和微分对策的事件触发自适应动态规划算法

针对一类非线性零和微分对策问题,本文提出了一种事件触发自适应动态规划(event-triggered adaptive dynamic programming,ET--ADP)算法在线求解其鞍点.首先,提出一个新的自适应事件触发条件.然后,利用一个输入为采样数据的神经网络(评价网络)近似最优值函数,并设计了新型的神经网络权值更新律使得值函数、控制策略及扰动策略仅在事件触发时刻同步更新.进一步地,利用Lyapunov稳定性理论证明了所提出的算法能够在线获得非线性零和微分对策的鞍点且不会引起Zeno行为.所提出的ET--ADP算法仅在事件触发条件满足时才更新值函数、控制策略和扰动策略,因而可有效减少计算量和降低网络负荷.最后,两个仿真例子验证了所提出的ET--ADP算法的有效性.

基于DHP方法的PEM燃料电池优化控制器设计

质子交换膜燃料电池(PEMFC)系统具有明显的非线性和时变的特质,因此质子交换膜燃料电池的建模和优化控制问题是研究的重点。通过建立单体质子交换膜燃料电池的近似线性动态模型,并在此模型基础上,设计了基于双启发式动态规划(DHP)的质子交换膜燃料电池神经网络优化控制器。仿真结果表明,此近似线性模型有效地简化了非线性和时变的特质,在此模型基础上所设计的神经网络控制器具有更好的控制效果和控制精度。