基于优先采样模型的离线强化学习

Offline Reinforcement Learning Based on Prioritized Sampling Model

离线强化学习通过减小分布偏移实现了习得策略向行为策略的逼近,但离线经验缓存的数据分布往往会直接影响习得策略的质量.通过优化采样模型来改善强化学习智能体的训练效果,提出两种离线优先采样模型:基于时序差分误差的采样模型和基于鞅的采样模型.基于时序差分误差的采样模型可以使智能体更多地学习值估计不准确的经验数据,通过估计更准确的值函数来应对可能出现的分布外状态.基于鞅的采样模型可以使智能体更多地学习对策略优化有利的正样本,减少负样本对值函数迭代的影响.进一步,将所提离线优先采样模型分别与批约束深度Q学习(Batch-constrained deep Q-learning,BCQ)相结合,提出基于时序差分误差的优先BCQ和基于鞅的优先BCQ.D4RL和Torcs数据集上的实验结果表明:所提离线优先采样模型可以有针对性地选择有利于值函数估计或策略优化的经验数据,获得更高的回报.

Offline reinforcement learning algorithms realize the approximation of learned policy to behavior policy by reducing the distribution shift,but the data distribution of offline experience buffer often directly affects the quality of learned policy.In this paper,two offline prioritized sampling models including temporal difference error-based and martingale-based are proposed to improve the training effect of reinforcement learning agent.The tem-poral difference error-based sampling model enables agents to learn more experience data with inaccurate value es-timation,thus deals with possible out-of-distribution states by estimating more accurate value functions.The mar-tingale-based sampling model enables agents to learn more positive samples beneficial to policy optimization and re-duces the impact of negative samples on value function iteration.Furthermore,the proposed offline prioritized sampling models are combined with the batch-constrained deep Q-learning(BCQ)respectively,to propose tempor-al difference error-based prioritized BCQ and martingale-based prioritized BCQ.Experimental results on D4RL and Torcs datasets show that the proposed two offline prioritized sampling models can be targeted to select the experi-ence data that are conducive to value function estimation or policy optimization,so as to obtain higher rewards.