基于最大熵强化学习的自主船舶航迹跟踪研究

Research on autonomous ship track tracking based on maximum entropy reinforcement learning

为解决自主船舶在航迹跟踪过程中使用最大熵强化学习作为控制器出现的收敛速度慢和训练时间长等问题,提出一种基于改进最大熵强化学习的航迹跟踪算法,引入了优先经验回放(PER)技术,并结合视线制导算法(LOS),构建PER-SAC的深度强化学习控制器,设计了相应的状态、动作空间和奖励函数。仿真结果表明,设计的PER-SAC控制器能快速收敛,收敛稳定后的控制器相较于原始SAC控制器控制性能更稳定,且控制精度更高,为自主船舶的航迹跟踪控制提供了一定参考价值。

To solve the problems of slow convergence and long training time that can occur when using maximum entropy reinforcement learning as a controller in the course of track tracking for autonomous ships,a track tracking algorithm based on improved maximum entropy reinforcement learning is proposed,introducing the preferred experience playback(PER)technique and combining it with the line of sight guidance algorithm(LOS)to construct a deep reinforcement learning controller for PER-SAC and design the corresponding the state,action space and reward function.Simulation results show that the designed PER-SAC controller can converge quickly,and the control performance is more stable and the control accuracy is higher after convergence and stabilisation compared to the original SAC controller,which provides some reference value for the track tracking control of autonomous ships.