参数未知的离散系统Q-学习优化状态估计与控制

Q-learning optimal state estimation and control for discrete systems with unknown parameters

控制系统的应用中存在状态不能直接测量或测量成本高的实际问题,给模型参数未知的系统完全利用状态数据学习最优控制器带来挑战性难题.为解决这一问题,首先构建具有状态观测器且系统矩阵中存在未知参数的离散线性增广系统,定义性能优化指标;然后基于分离定理、动态规划以及Q-学习方法,给出一种具有未知模型参数的非策略Q-学习算法,并设计近似最优观测器,得到完全利用可测量的系统输出和控制输入数据的非策略Q-学习算法,实现基于观测器状态反馈的系统优化控制策略,该算法的优点在于不要求系统模型参数全部已知,不要求系统状态直接可测,利用可测量数据实现指定性能指标的优化;最后,通过仿真实验验证所提出方法的有效性.

In the application of control systems, there is a practical problem that the state cannot be directly measured or the measurement cost is high. In order to solve this problem, a linear discrete-time augmented system with unknown parameters and a state observer is first constructed and the prescribed performance index is defined. Then, based on the separation theorem, the dynamic programming theory and the Q-learning method, a novel off-policy Q-learning algorithm is developed to approximate the optimal observer and the optimal controller for systems with unknown parameters and unmeasured states, such that the control performance is minimized using only measured data. The advantage of this algorithm is that it does not require all the system model parameters to be known and the system state to be directly measurable. Finally, the simulation experiment verifies the effectiveness of the proposed method.