基于改进强化学习的多无人机协同对抗算法研究

Research on multi-UAV cooperative confrontation algorithm based on improved reinforcement learning

多无人机的作战协同研究内容主要包含飞行协同、侦察协同及干扰协同,随着无人机数量及协同决策内容的增加,多智能体强化学习模型的状态空间及动作空间维度呈指数增长,多智能体强化学习算法在训练中不易收敛,协同决策水平难以得到显著提升。采用并对多智能体深度确定性策略梯度(MADDPG)算法原理进行模型构建,在此基础上提出了一种选择性经验存储策略的多智能体深度确定性策略梯度(SES-MADDPG)算法。该算法通过设置回收存储标准以及选择性因子,对进入经验池的经验进行选择性存储,以缓解奖励稀疏的问题。仿真实验表明,在保证算法时间复杂度的前提下,SES-MADDPG算法比其他强化学习算法有了更好的收敛效果,相较于MADDPG算法,任务完成率提高了25.427%。

The research of combat cooperation of multi-UAVs mainly includes flight cooperation,reconnaissance cooperation and interference cooperation.With the increase of both the number of UAVs and the content of cooperative decisions,state space and action space dimensions of the multi-agent reinforcement learning model grow exponentially.Multi-agent reinforcement learning algorithm is not easy to converge in training,and the level of cooperative decision-making is difficult to be significantly improved.This paper adopts and models on the principle of multi-agent deep deterministic policy gradient(MADDPG)algorithm,based on which it also proposes a multi-agent deep deterministic policy gradient algorithm of the selective experience storage policy(SES-MADDPG).The algorithm selectively stores the experience entering the experience pool by setting the recycling storage criteria as well as selectivity factors to alleviate the problem of reward sparsity.The simulation experiments show that,with guaranteed time complexity of the algorithm,the SES-MADDPG algorithm has a better convergence effect than other reinforcement learning algorithms,and shows an increase of 25.427%in task completion rate compared with MADDPG algorithm.