基于神经网络状态估计器的高速AUV强化学习控制

Reinforcement-Learning Control for the High-Speed AUV Based on the Neural-Network State Estimator

随着海洋研究与开发的日益扩大,高速自主水下航行器(AUV)作为重要的无人水下工作平台受到广泛关注。然而由于其模型具有多输入多输出、强耦合欠驱动以及强非线性特性,因此依赖精确模型的传统控制方法在实际应用中常受到限制。针对此问题,文中提出一种不依赖精确模型的强化学习位姿控制器,该控制器通过姿态环和位置环的配合不仅可以实现高速AUV的快速姿态稳定,还可以更快地完成下潜到指定深度的动作;同时,为了降低获取用于训练强化学习控制器数据的成本,结合神经网络技术提出了一种改进的高速AUV状态估计器,该估计器可以在已知当前时刻AUV的状态以及所受控制量的情况下估计出下一时刻的状态,从而为强化学习控制方法提供大量的训练数据。仿真实验结果表明,估计器达到了较高的估计精度,基于神经网络状态估计器训练得到的强化学习控制器可以完成原AUV的平稳快速控制,从而验证了所提方法的可行性及有效性。

With the development of ocean research and exploitation,high-speed autonomous undersea vehicle(AUV)has attracted increasing attention as important unmanned underwater platforms.However,the high-speed AUV model is multi-input-multi-output(MIMO),strong-coupling,underactuated,and strongly nonlinear;therefore,the traditional control method that relies on the exact model is often limited in practical applications.To address these problems,a position-attitude controller based on reinforcement learning(RL)that does not rely on an exact model is proposed.The RL controller can not only regulate the attitude of the AUV but also the driver,as it reaches the target depth faster with the aid of the attitude and position loops.An improved state estimator of a high-speed AUV is designed based on a neural network(NN)to decrease the cost of collecting data,which is employed to train the RL controller.The improved state estimator can estimate the state at the next time instant according to the current state of the high-speed AUV and the control input.The simulation results demonstrate that the NN-state-estimator can estimate the state of a high-speed AUV with high precision,and the RL controller trained by the estimator achieves fast and steady performance,which verifies the feasibility and effectiveness of the proposed method..