稳定且受限的新强化学习SAC算法

Novel Reinforcement Learning Algorithm:Stable Constrained Soft Actor Critic

为解决由于固定温度SAC(Soft Actor Critic)算法中存在的Q函数高估可能会导致算法陷入局部最优的问题,通过深入分析提出了一个稳定且受限的SAC算法(SCSAC:Stable Constrained Soft Actor Critic)。该算法通过改进最大熵目标函数修复固定温度SAC算法中的Q函数高估问题,同时增强算法在测试过程中稳定性的效果。最后,在4个OpenAI Gym Mujoco环境下对SCSAC算法进行了验证,实验结果表明,稳定且受限的SAC算法相比固定温度SAC算法可以有效减小Q函数高估出现的次数并能在测试中获得更加稳定的结果。

To solve the problem that Q function overestimation may cause SAC(Soft Actor Critic)algorithm trapped in local optimal solution,SCSAC(Stable Constrained Soft Actor Critic)algorithm is proposed for perfectly resolving the above weakness hidden in maximum entropy objective function improving the stability of Stable Constrained Soft Actor Critic algorithm in trailing process.The result of evaluating Stable Constrained Soft Actor Critic algorithm on the suite of OpenAI Gym Mujoco environments shows less Q value overestimation appearance and more stable results in trailing process comparing with SAC algorithm.