联合随机性策略的深度强化学习探索方法

Efficient exploration with stochastic policy for deep reinforcement learning

目前深度强化学习算法已经可以解决许多复杂的任务,然而如何平衡探索和利用的关系仍然是强化学习领域的一个基本的难题,为此提出一种联合随机性策略的深度强化学习探索方法。该方法利用随机性策略具有探索能力的特点,用随机性策略生成的经验样本训练确定性策略,鼓励确定性策略在保持自身优势的前提下学会探索。通过结合确定性策略算法DDPG和提出的探索方法,得到基于随机性策略指导的确定性策略梯度算法(SGDPG)。在多个复杂环境下的实验表明,面对探索问题,SGDPG的探索效率和样本利用率要优于DDPG算法。

At present,deep reinforcement learning algorithms have been shown to solve many complex tasks,but how to balance the relationship between exploration and exploitation is still a basic problem.Thus,this paper proposes an efficient exploration strategy combined with stochastic policy for deep reinforcement learning.The main contribution is to use the experience generated by stochastic policies to train deterministic policies,which encourages deterministic strategies to learn to explore while maintaining their own advantages.This takes advantage of the exploration ability of stochastic policies.By combining DDPG(Deep Deterministic Policy Gradient)and the proposed exploration method,the algorithm called stochastic guidance for deterministic policy gradient(SGDPG)is obtained.Finally,the results of the experiment in several complex environments show that SGDPG has higher exploration and sample efficiency than DDPG when faced with deep exploration problems.