基于最大熵深度强化学习的双足机器人步态控制方法

Gait control method based on maximum entropy deep reinforcement learning for biped robot

针对双足机器人连续直线行走的步态稳定控制问题,提出一种基于最大熵深度强化学习(DRL)的柔性演员-评论家(SAC)步态控制方法。首先,该方法无需事先建立准确的机器人动力学模型,所有参数均来自关节角而无需额外的传感器;其次,采用余弦相似度方法对经验样本分类,优化经验回放机制;最后,根据知识和经验设计奖励函数,使双足机器人在直线行走训练过程中不断进行姿态调整,确保直线行走的鲁棒性。在Roboschool仿真环境中与其他先进深度强化学习算法,如近端策略优化(PPO)方法和信赖域策略优化(TRPO)方法的实验对比结果表明,所提方法不仅实现了双足机器人快速稳定的直线行走,而且鲁棒性更好。

For the problem of gait stability control for continuous linear walking of a biped robot,a Soft Actor-Critic(SAC)gait control algorithm based on maximum entropy Deep Reinforcement Learning(DRL)was proposed.Firstly,without accurate robot dynamic model built in advance,all parameters were derived from joint angles without additional sensors.Secondly,the cosine similarity method was used to classify experience samples and optimize the experience replay mechanism.Finally,reward functions were designed based on knowledge and experience to enable the biped robot continuously adjust its attitude during the linear walking training process,and the reward functions ensured the robustness of straight walking.The proposed method was compared with other DRL methods such as PPO(Proximal Policy Optimization)and TRPO(Trust Region Policy Optimization)in Roboschool simulation environment.The results show that the proposed method not only achieves fast and stable linear walking of the biped robot,but also has better algorithmic robustness.