基于值函数估计的参数探索策略梯度算法

Function approximation for policy gradients with parameter-based exploration

策略梯度估计方差大是策略梯度算法存在的普遍问题,基于参数探索的策略梯度算法(PGPE)通过使用确定性策略有效缓解了这一问题。然而,PGPE算法基于蒙特卡罗方法进行策略梯度的估计,需要大量学习样本才能保证梯度估计相对稳定,因此,梯度估计方差大阻碍了其在现实问题中的实际应用。为进一步减小PGPE算法策略梯度估计的方差,提出了基于值函数估计的参数探索策略梯度算法(PGPE-FA),该算法在PGPE算法中引入Actor-Critic框架。具体地,提出的方法使用价值函数估计策略梯度,代替了PGPE方法使用轨迹样本估计策略梯度的方式,从而减小了梯度估计方差。最后,通过实验验证了所提算法能够减小梯度估计的方差。

Policy gradient algorithms suffer from the large variance of gradient estimation.the algorithm of policy gradient with parameter based exploration mitigates this problem to some extent.However,PGPE estimates its gradient based on the Monte Carlo,which requires a large number of samples to achieve the fairly stable policy update.And thus hinders its application in the real world problem.In order to further reduce the variance of policy gradient,the algorithm of function approximation for policy gradients with parameter-based exploration(PGPE-FA)implements the algorithm of PGPE in the Actor-Critic framework.More specifically,the proposed method utilized value function to estimate the policy gradient,instead of using trajectory samples to estimate the policy gradient as PGPE method does,thereby reducing the variance of gradient estimation.Finally,the experiment verifies that the proposed algorithm can reduce the variance of gradient estimation.