一种用于LQR控制问题的强化学习方法

A Reinforcement Learning Method for LQR Control Problem

现有强化学习方法的收敛性分析大多针对离散状态问题,对于连续状态问题强化学习的收敛性分析仅局限于简单的 LQR 控制问题.本文对现有两种用于 LQR 问题收敛的强化学习方法进行分析,针对存在的问题,提出一种只需部分模型信息的强化学习方法.该方法使用递推最小二乘 TD(RLS-TD)方法估计值函数参数,递推最小二乘方法(RLS)估计贪心改进策略.并给出理想情况下此方法收敛的理论分析.仿真实验表明该方法收敛到最优控制策略.

Current convergence analyses of reinforcement learning method are mainly applied to discrete state problems. Analyses of continuous state reinforcement learning method are limited to simple LQR control problems. After analyzing two convergent reinforcement learning methods for LQR control problem, a new method only requiring partial model information is proposed to make up for the defects of these two methods. In this method, a recursive least-squares TD method is used to estimate parameters of value function and a recursive least-squares method is used to estimate the greedily improved policy . In theoretical analysis , a convergence proof is presented for the proposed policy iteration method in ideal case. Simulation result shows that this method converges an optimal control policy.