融合自适应评判的随机系统数据驱动策略优化

Data-driven Policy Optimization for Stochastic Systems Involving Adaptive Critic

自适应评判技术已经广泛应用于求解复杂非线性系统的最优控制问题,但利用其求解离散时间非线性随机系统的无限时域最优控制问题还存在一定局限性.本文融合自适应评判技术,建立一种数据驱动的离散随机系统折扣最优调节方法.首先,针对宽松假设下的非线性随机系统,研究带有折扣因子的无限时域最优控制问题.所提的随机系统Q-learning算法能够将初始的容许策略单调不增地优化至最优策略.基于数据驱动思想,随机系统Q-learning算法在不建立模型的情况下直接利用数据进行策略优化.其次,利用执行−评判神经网络方案,实现了随机系统Q-learning算法.最后,通过两个基准系统,验证本文提出的随机系统Q-learning算法的有效性.

Adaptive critic technology has been widely employed to solve the optimal control problems of complicated nonlinear systems,but there are some limitations to solve the infinite-horizon optimal problems of discrete-time nonlinear stochastic systems.In this paper,we establish a data-driven discounted optimal regulation method for discrete-time stochastic systems involving adaptive critic technology.First,we investigate the infinite-horizon optimal problems with the discount factor for stochastic systems under the relaxed assumption.The developed stochastic Qlearning algorithm can optimize an initial admissible policy to the optimal one in a monotonically nonincreasing way.Based on the data-driven idea,the policy optimization of the stochastic Q-learning algorithm is executed without a dynamic model.Then,the stochastic Q-learning algorithm is implemented by utilizing the actor-critic neural networks.Finally,two nonlinear benchmarks are given to demonstrate the overall performance of the developed stochastic Q-learning algorithm.