基于多智能体深度强化学习的无人机动态预部署策略

A Dynamic Pre-Deployment Strategy of UAVs Based on Multi-Agent Deep Reinforcement Learning

针对传统优化算法在求解长时间尺度内通信无人机(UAV)动态部署时复杂度过高且难以与动态环境信息匹配等缺陷,该文提出一种基于多智能体深度强化学习(MADRL)的UAV动态预部署策略。首先利用一种深度时空网络模型预测用户的预期速率需求以捕捉动态环境信息,定义用户满意度的概念以刻画用户所获得UAV提供服务的公平性,并以最大化长期总体用户满意度和最小化UAV移动及发射能耗为目标建立优化模型。其次,将上述模型转化为部分可观测马尔科夫博弈过程(POMG),并提出一种基于MADRL的H-MADDPG算法求解该POMG中轨迹规划、用户关联和功率分配的最佳决策。该H-MADDPG算法使用混合网络结构以实现对多模态输入的特征提取,并采用集中式训练-分布式执行的机制以高效地训练和执行决策。最后仿真结果证明了所提算法的有效性。

It’s challenging to use traditional optimization algorithms to solve the long-term dynamic deployment problem of Unmanned Aerial Vehicles(UAVs)due to their high complexity and difficulty in matching dynamic environment.Aiming at solving these shortcomings,a dynamic pre-deployment strategy of UAV based on Multi-Agent Deep Reinforcement Learning(MADRL)is proposed.Firstly,a deep spatio-temporal network model is used to predict the expected rate demand of users in the coverage area to capture the dynamic environment information.The concept of users’satisfaction is defined to describe the fairness of users.An optimization problem is modeled with the goal of maximizing the long-term overall users’satisfaction,minimizing the mobile and radio energy consumption of the UAVs.Secondly,the problem above is transformed into a Partially Observable Markov Game(POMG)process.An H-MADDPG algorithm based on MADRL is proposed to solve the optimal decision of trajectory design,user association and power allocation.The HMADDPG algorithm uses a hybrid network structure to extract the features of multi-modal inputs,and adopts a centralized training-distributed execution mechanism to realize efficient training and decision execution.Finally,the effectiveness of the algorithm is verified by simulation experiments.