Policy Iteration Algorithms for Zero-Sum Stochastic Differential Games with Long-Run Average Payoff Criteria

This paper studies the policy iteration algorithm(PIA)for zero-sum stochastic differential games with the basic long-run average criterion,as well as with its more selective version,the so-called bias criterion.The system is assumed to be a nondegenerate diffusion.We use Lyapunov-like stability conditions that ensure the existence and boundedness of the solution to certain Poisson equation.We also ensure the convergence of a sequence of such solutions,of the corresponding sequence of policies,and,ultimately,of the PIA.

Adaptive dynamic programming for online solution of a zero-sum differential game

This paper will present an approximate/adaptive dynamic programming(ADP) algorithm,that uses the idea of integral reinforcement learning(IRL),to determine online the Nash equilibrium solution for the two-player zerosum differential game with linear dynamics and infinite horizon quadratic cost.The algorithm is built around an iterative method that has been developed in the control engineering community for solving the continuous-time game algebraic Riccati equation(CT-GARE),which underlies the game problem.We here show how the ADP techniques will enhance the capabilities of the offline method allowing an online solution without the requirement of complete knowledge of the system dynamics.The feasibility of the ADP scheme is demonstrated in simulation for a power system control application.The adaptation goal is the best control policy that will face in an optimal manner the highest load disturbance.

Nash and Stackelberg Differential Games

A large class of stochastic differential games for several players is considered in this paper.The class includes Nash differential games as well as Stackelberg differential games.A mix is possible.The existence of feedback strategies under general conditions is proved.The limitations concern the functionals in which the state and the controls appear separately.This is also true for the state equations.The controls appear in a quadratic form for the payoff and linearly in the state equation.The most serious restriction is the dimension of the state equation,which cannot exceed 2.The reason comes from PDE(partial differential equations) techniques used in studying the system of Bellman equations obtained by Dynamic Programming arguments.In the authors' previous work in 2002,there is not such a restriction,but there are serious restrictions on the structure of the Hamiltonians,which are violated in the applications dealt with in this article.

Three-dimensional nonlinear H_2/H_∞ guidance law based upon approach of solving the state feedback Nash balance point

Based upon the theory of the nonlinear quadric two-person nonzero-sum differential game,the fact that the time-limited mixed H2/H∞ control problem can be turned into the problem of solving the state feedback Nash balance point is mentioned. Upon this,a theorem about the solution of the state feedback control is given,the Lyapunov stabilization of the nonlinear system under this control is proved,too. At the same time,this solution is used to design the nonlinear H2/H∞ guidance law of the relative motion between the missile and the target in three-dimensional(3D) space. By solving two coupled Hamilton-Jacobi partial differential inequalities(HJPDI),a control with more robust stabilities and more robust performances is obtained. With different H∞ performance indexes,the correlative weighting factors of the control are analytically designed. At last,simulations under different robust performance indexes and under different initial conditions and under the cases of intercepting different maneuvering targets are carried out. All results indicate that the designed law is valid.

Solution of a zero-sum linear quadratic differential game with singular control cost of minimiser

We consider a finite horizon,zero-sum linear quadratic differential game.The feature of this game is that a weight matrix of the minimiser’s control cost in the cost functional is singular.Due to this singularity,the game can be solved neither by applying the Isaacs MinMax principle nor using the Bellman–Isaacs equation approach,i.e.this game is singular.Aprevious paper of one of the authors analysed such a game in the case where the cost functional does not contain the minimiser’s control cost at all,i.e.the weight matrix of this cost equals zero.In this case,all coordinates of the minimiser’s control are singular.In the present paper,we study the general case where the weight matrix of the minimiser’s control cost,being singular,is not,in general,zero.This means that only a part of the coordinates of the minimiser’s control is singular,while others are regular.The considered game is treated by a regularisation,i.e.by its approximate conversion to an auxiliary regular game.The latter has the same equation of dynamics and a similar cost functional augmented by an integral of the squares of the singular control coordinates with a small positive weight.Thus,the auxiliary game is a partial cheap control differential game.Based on a singular perturbation’s asymptotic analysis of this auxiliary game,the existence of the value of the original(singular)game is established,and its expression is obtained.The maximiser’s optimal state feedback strategy and the minimising control sequence in the original game are designed.It is shown that the coordinates of the minimising control sequence,corresponding to the regular coordinates of the minimiser’s control,are point-wise convergent in the class of regular functions.The optimal trajectory sequence and the optimal trajectory in the considered singular game also are obtained.An illustrative example is presented.

Optimal Strategy for Aircraft Pursuit-evasion Games via Self-play Iteration

In this paper,the pursuit-evasion game with state and control constraints is solved to achieve the Nash equilibrium of both the pursuer and the evader with an iterative self-play technique.Under the condition where the Hamiltonian formed by means of Pontryagin’s maximum principle has the unique solution,it can be proven that the iterative control law converges to the Nash equilibrium solution.However,the strong nonlinearity of the ordinary differential equations formulated by Pontryagin’s maximum principle makes the control policy difficult to figured out.Moreover the system dynamics employed in this manuscript contains a high dimensional state vector with constraints.In practical applications,such as the control of aircraft,the provided overload is limited.Therefore,in this paper,we consider the optimal strategy of pursuit-evasion games with constant constraint on the control,while some state vectors are restricted by the function of the input.To address the challenges,the optimal control problems are transformed into nonlinear programming problems through the direct collocation method.Finally,two numerical cases of the aircraft pursuit-evasion scenario are given to demonstrate the effectiveness of the presented method to obtain the optimal control of both the pursuer and the evader.

Computational intelligence interception guidance law using online off-policy integral reinforcement learning

Missile interception problem can be regarded as a two-person zero-sum differential games problem,which depends on the solution of Hamilton-Jacobi-Isaacs(HJI)equa-tion.It has been proved impossible to obtain a closed-form solu-tion due to the nonlinearity of HJI equation,and many iterative algorithms are proposed to solve the HJI equation.Simultane-ous policy updating algorithm(SPUA)is an effective algorithm for solving HJI equation,but it is an on-policy integral reinforce-ment learning(IRL).For online implementation of SPUA,the dis-turbance signals need to be adjustable,which is unrealistic.In this paper,an off-policy IRL algorithm based on SPUA is pro-posed without making use of any knowledge of the systems dynamics.Then,a neural-network based online adaptive critic implementation scheme of the off-policy IRL algorithm is pre-sented.Based on the online off-policy IRL method,a computa-tional intelligence interception guidance(CIIG)law is developed for intercepting high-maneuvering target.As a model-free method,intercepting targets can be achieved through measur-ing system data online.The effectiveness of the CIIG is verified through two missile and target engagement scenarios.

基于微分对策理论的非线性控制回顾与展望

微分对策是使用微分方程处理双方或多方连续动态冲突、竞争或合作问题的一种数学工具.它已经广泛应用于生物学、经济学、国际关系、计算机科学和军事战略等诸多领域.微分对策实质上是一种双方或多方的最优控制问题,它将现代控制理论与对策论相融合,从而比控制理论具有更强的竞争性、对抗性和适用性.本文根据非线性微分对策理论的控制、均衡及算法阐述了微分对策的理论发展历史,综述了已有结论与算法的本质,总结了现有的研究成果.最后对基于微分对策理论非线性系统的鲁棒性与最优性进行了展望.

基于统计线性化的随机非线性微分对策逼近最优策略

针对二人零和随机非线性微分对策问题,利用统计线性化技术并提出一种新的逼近策略控制方法.通过求解具有统计线性化参数的Riccati微分方程得到逼近控制策略,该Riccati微分方程与一般线性系统的Riccati微分方程具有明显的区别;同时对控制量受约束情形的控制策略也进行了求解;最后通过仿真验证了所得结论的正确性.

追捕逃逸型微分对策问题的识别域判别

研究了仿射非线性控制系统下的单目标两人追捕逃逸型微分对策问题,解决了该类控制系统在不等式约束区域上系统识别域的判别问题,给出了判别识别域的充分必要条件.首先利用生存理论及非光滑分析工具得到了仿射非线性控制系统的识别域判别定理,从而把对该非线性控制系统识别域的判别问题转化为求解凸不等式组的相容性问题.基于凸可行问题的求解方法给出了此问题的投影算法,并给出算法相应的收敛性定理.最后得到了仿射非线性系统下的两人追捕逃逸型微分对策问题的选择定理.

非线性约束系统H_∞鲁棒预测控制

针对一类具有状态约束和控制约束的离散非线性系统,本文采用有限维参数化方法提出了一种基于闭环优化的H∞鲁棒预测控制算法.这种算法把预测控制的滚动优化机制同微分对策理论和非线性H∞控制理论做了有机结合;在闭环优化中通过有限维参数化方法把控制变量参数化为多项式控制变量,并且引入被控系统的过渡平衡点.这样算法不但可以处理不确定系统,而且降低了在线闭环优化的计算复杂度.进一步,证明了算法的可行性和对有界不确定性系统的鲁棒稳定性.最后,用数值仿真验证了算法的有效性.

基于SDRE具有终端碰撞角约束的非线性微分对策制导律

通过非线性微分对策理论讨论了具有终端碰撞角约束的弹目追逃问题,提出了基于求解状态相关黎卡提方程(state-dependent Riccati equation,SDRE)的方法解决该微分对策问题,得到了具有终端碰撞角约束的SDRE解析解。提出的终端碰撞角约束制导律是以弹目视线角速率及碰撞角误差为状态向量进行推导,并研究了闭环系统局部渐进稳定的条件。该制导律不需要进行剩余时间的预测。最后针对目标不进行机动、进行阶跃机动、正弦机动及目标最优机动形式4种情况,进行了制导律的仿真验证,仿真结果表明该制导律对于不同机动目标均具有良好的制导效果且能很好地满足末端碰撞角约束要求。

非线性零和微分对策的事件触发自适应动态规划算法

针对一类非线性零和微分对策问题,本文提出了一种事件触发自适应动态规划(event-triggered adaptive dynamic programming,ET--ADP)算法在线求解其鞍点.首先,提出一个新的自适应事件触发条件.然后,利用一个输入为采样数据的神经网络(评价网络)近似最优值函数,并设计了新型的神经网络权值更新律使得值函数、控制策略及扰动策略仅在事件触发时刻同步更新.进一步地,利用Lyapunov稳定性理论证明了所提出的算法能够在线获得非线性零和微分对策的鞍点且不会引起Zeno行为.所提出的ET--ADP算法仅在事件触发条件满足时才更新值函数、控制策略和扰动策略,因而可有效减少计算量和降低网络负荷.最后,两个仿真例子验证了所提出的ET--ADP算法的有效性.

一类非线性不确定系统鲁棒H^∞控制器的设计

利用微分对策方法 ,讨论了一类不确定性非线性控制系统的鲁棒H∞ 控制问题。给出了在所有允许的不确定范围内 ,使闭环系统具有鲁棒H∞ 控制特性的状态反馈鲁棒H∞ 控制器、输出反馈鲁棒H∞ 控制器以及基于观测器的鲁棒H∞ 控制的设计方法。指出了如果相应的一个或两个Hamilton -Jacobi不等式有非负解 ,则该不确定非线性系统的鲁棒H∞ 控制问题有解。

有界扰动下约束非线性系统鲁棒经济模型预测控制

针对未知但有界扰动下约束非线性系统,提出一种新的鲁棒经济模型预测控制(Economic model predictive control,EMPC)策略,保证闭环系统对扰动输入具有输入到状态稳定性(Input-to-state stability,ISS).基于微分对策原理,分别优化经济目标函数和关于最优经济平衡点的鲁棒稳定性目标函数,其中经济最优性与鲁棒稳定性是具有冲突的两个控制目标.利用鲁棒稳定性目标最优值函数构造EMPC优化的隐式收缩约束,建立鲁棒EMPC的递推可行性和闭环系统关于最优经济平衡点相对于有界扰动输入到状态稳定性结果.最后以连续搅拌反应器为例,对比仿真验证本文策略的有效性.

基于有限可解群的一体化电力调度自动化系统

研究了一体化电力调度自动化系统以及有限可解群对一体化电力调度影响的基础上,提出一种融合有限可解群(FSG)的一体化电力调度自动化系统方法。通过使用有限可解群算法在一体化电力调度自动化系统领域的应用。同时,在研究一体化电力调度自动化系统的基本原理上,设计有限可解群在两相旋转坐标系下的数学模型,引入有限可解群算法方法,可以通过调节励磁来独立调节一体化电力调度自动化系统的有功与无功。实验结果表明,该方法对一体化电力调度自动化系统具有比较好的控制效果,能够降低一体化电力调度自动化系统的磨损程度。

非线性系统 H_∞ 的输出反馈鲁棒镇定

研究了目前有共性的几种非线性系统输出反馈Ｈ∞干扰衰减方法，证明输出反馈干扰衰减可解的充分条件，给出了状态观测器的具体形式，从实用性和鲁棒性两方面发展了目前非线性系统Ｈ∞输出反馈鲁棒控制的结果．文中主要采用微分对策和Ｌｙａｐｕｎｏｖ方法建立非线性系统Ｈ∞状态反馈干扰衰减可解的充分条件（Ｈａｍｉｌｔｏｎ－Ｊａｃｏｂｉ－Ｉｓａａｃｓ偏微分方程），并结合复合Ｌｙａｐｕｎｏｖ函数构造输出反馈Ｈ∞干扰衰减Ｈａｍｉｌｔｏｎ－Ｊａｃｏｂｉ－Ｉｓａａｃｓ偏微分方程．研究了这一输出反馈律对闭环系统的鲁棒性，并证明了相应闭环系统渐近稳定的充分条件．

基于网络动态交通路由问题的非线性H_∞控制(英文)

为了优化交通网络的动态路由,运用非线性H∞控制方法和简单的实时代数计算方法,使系统在有界干扰情况下具有鲁棒性,避免了现有方法中冗长的离线或在线数学计算,解决了网络层面上系统最优和用户平衡的DIA(dynamic traffic assignment)/DTR(dynamic traffic routing)问题.该方法适用于解决先进交通信息管理系统(ATMIS)中的DTA/DTR问题.

基于自适应最优控制的有限时间微分对策制导律

针对固定末端时刻拦截机动目标的制导系统,本文首先构建了非线性有限时间微分对策框架,将导弹拦截非线性系统的最优问题转化为一般非线性系统的最优控制问题,并通过自适应动态规划算法(adaptive dynamic programming,ADP)获得近似最优值函数与最优控制策略.为了有效实现该算法,本文利用一个具有时变权值和激活函数的评价网络来逼近Hamilton-Jacobi-Isaacs(HJI)方程的解,并在线更新.通过李雅普诺夫法来证明本文提出的控制策略可保证闭环微分对策系统稳定性和评价网络权值近似误差的有界性.最后给出一个非线性导弹拦截目标系统的仿真例子验证了该方法的可行性和有效性.

基于加权果蝇优化算法的多区域频率协同控制

随着大规模可再生能源的开发和应用,电网变得越来越庞大且复杂,如何保证大量不同控制器之间的协调是最值得关注的问题之一。利用微分博弈理论可以解决协同控制的问题。然而,传统算法难以求解带约束的微分博弈问题。此外,现有研究建立的仿真模型几乎是线性的,不利于实际工程应用。针对上述问题,提出了一种基于加权果蝇优化算法(Weighting Fruit Fly Optimization Algorithm,WFOA)的协同进化算法来求解具有非线性约束的多区域频率协同控制模型。仿真结果表明,与协同进化遗传算法和协同多目标粒子群优化算法相比,该方法具有更好的控制效率,同时对系统出现的外部扰动变化及内部机组参数变动具有很好的鲁棒性。