基于优化并行的四足机器人运动技能学习

Optimization-based parallel learning of quadruped robot locomotion skills

动物对自然界的适应能力是由环境选择与适者生存决定的,四足哺乳动物可以通过种群的进化逐步适应环境的变化,提高其对环境的适应度和种群的生存率。基于上述启发,该文在软演员-评论家(SAC)算法基础上提出一种基于优化并行强化学习的算法OP-SAC,该算法使用进化策略与强化学习交替训练,通过知识共享和知识继承优化四足机器人学习效果,提高训练效率。算法验证结果显示,OP-SAC算法能够完成四足机器人的仿生步态学习;对比实验验证出OP-SAC算法比其他结合了进化策略的SAC算法具有更加优越的鲁棒性;设计消融实验证明了知识共享和知识继承策略使算法的训练效果获得较大提升。

[Objective]Inspired by the skill learning of quadruped animals in nature,deep reinforcement learning has been widely applied to learn the quadruped robot locomotion skill.Through interaction with the environment,robots can autonomously learn complete motion strategies.However,traditional reinforcement learning has several drawbacks,such as large computational requirements,slow convergence of algorithms,and rigid learning strategies,which substantially reduce training efficiency and generate unnecessary time costs.To address these shortcomings,this paper introduces evolutionary strategies into the soft actor-critic(SAC)algorithm,proposing an optimized parallel SAC(OP-SAC)algorithm for the parallel training of quadruped robots using evolutionary strategies and reinforcement learning.[Methods]The algorithm first uses a variant temperature coefficient SAC algorithm to reduce the impact of hyperparameter temperature coefficients on the training process and then introduces evolutionary strategies using the reference trajectory trained by the evolutionary strategy as a sample input to guide the training direction of the SAC algorithm.Additionally,the state information and reward values obtained from SAC algorithm training are used as inputs and offspring selection thresholds for the evolutionary strategy,achieving the decoupling of training data.Furthermore,the algorithm adopts an alternating training approach,introducing a knowledge-sharing strategy where the training results of the evolutionary strategy and reinforcement learning are stored in a common experience pool.Furthermore,a knowledge inheritance mechanism is introduced,allowing the training results of both strategies to be passed on to the next stage of the algorithm.With these two training strategies,the evolutionary strategy and reinforcement learning can guide each other in terms of the training direction and pass useful information between different generations,thereby accelerating the learning process and enhancing the robustness of the algorithm.[Results]The simulation experiment results were as follows:1)Using the OP-SAC algorithm to train quadruped robots achieves a reward value convergence of approximately 3000,with stable posture and high speed after training completion.The algorithm can effectively complete the bionic gait learning of quadruped robots.2)Compared with other algorithms combining SAC and evolutionary strategies,the OP-SAC algorithm has a faster convergence rate and higher reward value after convergence,demonstrating higher robustness in the learned strategies.3)Although the convergence speed of the OP-SAC algorithm is slower than that of other reinforcement learning algorithms combined with central pattern generator,it ultimately achieves a higher reward value and more stable training results.4)The ablation experiments confirm the importance of knowledge inheritance and sharing strategies for improving training effectiveness.[Conclusions]The above analysis shows that the proposed OP-SAC algorithm accomplishes the learning of quadruped robot locomotion skill,improves the convergence speed of reinforcement learning to a certain extent,optimizes learning strategies,and significantly enhances training efficiency.