样本有限关联值递归Q学习算法及其收敛性证明

A RELATIVE VALUE ITERATION Q-LEARNING ALGORITHM AND ITS CONVERGENCE BASED-ON FINITE SAMPLES

一个激励学习 Agent通过学习一个从状态到动作映射的最优策略来解决策问题 .求解最优决策一般有两种途径 :一种是求最大奖赏方法 ,另一种是求最优费用方法 .利用求解最优费用函数的方法给出了一种新的 Q学习算法 .Q学习算法是求解信息不完全 Markov决策问题的一种有效激励学习方法 .Watkins提出了 Q学习的基本算法 ,尽管他证明了在满足一定条件下 Q值学习的迭代公式的收敛性 ,但是在他给出的算法中 ,没有考虑到在迭代过程中初始状态与初始动作的选取对后继学习的影响 .因此提出的关联值递归 Q学习算法改进了原来的 Q学习算法 ,并且这种算法有比较好的收敛性质 .从求解最优费用函数的方法出发 ,给出了 Q学习的关联值递归算法 ,这种方法的建立可以使得动态规划 (DP)算法中的许多结论直接应用到 Q学习的研究中来 .

A reinforcement learning agent solves its decision problems by learning optimal decision mapping from a state to an action. There are two approaches generally to solving optimal decision, the one for maximum reward, the other for optimal cost. This paper is concerned with the problem of a novel Q-learning algorithm for solving optimal cost function. Q-learning is a reinforcement learning method to solve Markovian decision problems with incomplete information. In this paper, beginning with solving optimality cost function, the relative value iteration Q-learning algorithm is proposed. It can make many results of the dynamics programming algorithm available for studying Q-learning directly.