Cooperative decision-making algorithm with efficient convergence for UCAV formation in beyond-visual-range air combat based on multi-agent reinforcement learning

Highly intelligent Unmanned Combat Aerial Vehicle(UCAV)formation is expected to bring out strengths in Beyond-Visual-Range(BVR)air combat.Although Multi-Agent Reinforcement Learning(MARL)shows outstanding performance in cooperative decision-making,it is challenging for existing MARL algorithms to quickly converge to an optimal strategy for UCAV formation in BVR air combat where confrontation is complicated and reward is extremely sparse and delayed.Aiming to solve this problem,this paper proposes an Advantage Highlight Multi-Agent Proximal Policy Optimization(AHMAPPO)algorithm.First,at every step,the AHMAPPO records the degree to which the best formation exceeds the average of formations in parallel environments and carries out additional advantage sampling according to it.Then,the sampling result is introduced into the updating process of the actor network to improve its optimization efficiency.Finally,the simulation results reveal that compared with some state-of-the-art MARL algorithms,the AHMAPPO can obtain a more excellent strategy utilizing fewer sample episodes in the UCAV formation BVR air combat simulation environment built in this paper,which can reflect the critical features of BVR air combat.The AHMAPPO can significantly increase the convergence efficiency of the strategy for UCAV formation in BVR air combat,with a maximum increase of 81.5%relative to other algorithms.

HEURISTIC PARTICLE SWARM OPTIMIZATION ALGORITHM FOR AIR COMBAT DECISION-MAKING ON CMTA

Combining the heuristic algorithm （HA） developed based on the specific knowledge of the cooperative multiple target attack （CMTA） tactics and the particle swarm optimization （PSO）, a heuristic particle swarm optimization （HPSO） algorithm is proposed to solve the decision-making （DM） problem. HA facilitates to search the local optimum in the neighborhood of a solution, while the PSO algorithm tends to explore the search space for possible solutions. Combining the advantages of HA and PSO, HPSO algorithms can find out the global optimum quickly and efficiently. It obtains the DM solution by seeking for the optimal assignment of missiles of friendly fighter aircrafts （FAs） to hostile FAs. Simulation results show that the proposed algorithm is superior to the general PSO algorithm and two GA based algorithms in searching for the best solution to the DM problem.

Air combat decision-making of multiple UCAVs based on constraint strategy games

Game theory can be applied to the air combat decision-making problem of multiple unmanned combat air vehicles(UCAVs).However,it is difficult to have satisfactory decision-making results completely relying on air combat situation information,because there is a lot of time-sensitive information in a complex air combat environment.In this paper,a constraint strategy game approach is developed to generate intelligent decision-making for multiple UCAVs in complex air combat environment with air combat situation information and time-sensitive information.Initially,a constraint strategy game is employed to model attack-defense decision-making problem in complex air combat environment.Then,an algorithm is proposed for solving the constraint strategy game based on linear programming and linear inequality(CSG-LL).Finally,an example is given to illustrate the effectiveness of the proposed approach.

Multi-target Collaborative Combat Decision-Making by Improved Particle Swarm Optimizer

A decision-making problem of missile-target assignment with a novel particle swarm optimization algorithm is proposed when it comes to a multiple target collaborative combat situation.The threat function is established to describe air combat situation.Optimization function is used to find an optimal missile-target assignment.An improved particle swarm optimization algorithm is utilized to figure out the optimization function with less parameters,which is based on the adaptive random learning approach.According to the coordinated attack tactics,there are some adjustments to the assignment.Simulation example results show that it is an effective algorithm to handle with the decision-making problem of the missile-target assignment(MTA)in air combat.

Optimal Formation Reconfiguration Control of Multiple UCAVs Using Improved Particle Swarm Optimization

Optimal formation reconfiguration control of multiple Uninhabited Combat Air Vehicles （UCAVs） is a complicated global optimum problem. Particle Swarm Optimization （PSO） is a population based stochastic optimization technique inspired by social behaviour of bird flocking or fish schooling. PSO can achieve better results in a faster, cheaper way compared with other bio-inspired computational methods, and there are few parameters to adjust in PSO. In this paper, we propose an improved PSO model for solving the optimal formation reconfiguration control problem for multiple UCAVs. Firstly, the Control Parameterization and Time Diseretization （CPTD） method is designed in detail. Then, the mutation strategy and a special mutation-escape operator are adopted in the improved PSO model to make particles explore the search space more efficiently. The proposed strategy can produce a large speed value dynamically according to the variation of the speed, which makes the algorithm explore the local and global minima thoroughly at the same time. Series experimental results demonstrate the feasibility and effectiveness of the proposed method in solving the optimal formation reconfiguration control problem for multiple UCAVs.

AIRCRAFT CONCEPT EVALUATION AND EFFECTIVENESS-BASED DECISION-MAKING

Based on effectiveness analysis , a novel method is presented for combat aircraft top-hierarchy concept evaluation and decision-making. Applying multi-criterion decision-making ( MCDM ) and analytic hierarchy process , the new method can help to overcome the limitations of existing evaluation systems and decision-make methods.The proposed method includes the following process :( 1 ) Establish a multi-criterion and multi-hierarchy evaluation attribute system by introducing combat effectiveness ;( 2 ) Assign weight to the attributes and normalize them ;( 3 ) Evaluate and decision-make top-hierarchy aircraft concept based on effectiveness to reach a satisfactory design by comprehensively applying four multi-criterion decision-making methodologies , i.e.grey correlation projection method , weighted summation method , weighted quadrature method and ideal solution decision-making method , while considering the attribute hierarchy system and the logical relations among the attributes.Finally , an example is given to indicate the validity and feasibility of the proposed method.

Threat sequencing of multiple UCAVs with incomplete information based on game theory

The threat sequencing of multiple unmanned combat air vehicles(UCAVs) is a multi-attribute decision-making(MADM)problem. In the threat sequencing process of multiple UCAVs,due to the strong confrontation and high dynamics of the air combat environment, the weight coefficients of the threat indicators are usually time-varying. Moreover, the air combat data is difficult to be obtained accurately. In this study, a threat sequencing method of multiple UCAVs is proposed based on game theory by considering the incomplete information. Firstly, a zero-sum game model of decision maker( D) and nature(N)with fuzzy payoffs is established to obtain the uncertain parameters which are the weight coefficient parameters of the threat indicators and the interval parameters of the threat matrix. Then,the established zero-sum game with fuzzy payoffs is transformed into a zero-sum game with crisp payoffs(matrix game) to solve. Moreover, a decision rule is addressed for the threat sequencing problem of multiple UCAVs based on the obtained uncertain parameters. Finally, numerical simulation results are presented to show the effectiveness of the proposed approach.

UAV Maneuvering Decision-Making Algorithm Based on Twin Delayed Deep Deterministic Policy Gradient Algorithm

Aiming at intelligent decision-making of unmanned aerial vehicle(UAV)based on situation information in air combat,a novelmaneuvering decision method based on deep reinforcement learning is proposed in this paper.The autonomous maneuvering model ofUAV is established byMarkovDecision Process.The Twin DelayedDeep Deterministic Policy Gradient(TD3)algorithm and the Deep Deterministic Policy Gradient(DDPG)algorithm in deep reinforcement learning are used to train the model,and the experimental results of the two algorithms are analyzed and compared.The simulation experiment results show that compared with the DDPG algorithm,the TD3 algorithm has stronger decision-making performance and faster convergence speed and is more suitable for solving combat problems.The algorithm proposed in this paper enables UAVs to autonomously make maneuvering decisions based on situation information such as position,speed,and relative azimuth,adjust their actions to approach,and successfully strike the enemy,providing a new method for UAVs to make intelligent maneuvering decisions during air combat.

基于兵棋推演的空战编组对抗智能决策方法

基于兵棋研究的空战编组对抗方法主要使用规则或运筹等手段,存在假设不够合理、建模不准确、应变性差等缺陷。强化学习算法可以根据作战数据自主学习编组对抗策略,以应对复杂的战场情况,但现有强化学习对作战数据要求高,当动作空间过大时,算法收敛慢,且对仿真平台有较高的要求。针对上述问题,提出了一种融合知识数据和强化学习的空战编组对抗智能决策方法,该决策方法的输入是战场融合态势,使用分层决策框架控制算子选择并执行任务,上层包含使用专家知识驱动的动作选择器,下层包含使用专家知识和作战规则细化的避弹动作执行器、侦察动作执行器和使用强化学习算法控制的打击动作执行器。最后基于典型作战场景进行实验,验证了该方法的可行性和实用性,且具有建模准确、训练高效的优点。

基于动态一致性联盟算法的异构无人机集群协同作战联盟组建

动态未知乃至对抗条件下的异构无人机集群协同作战联盟组建是无人机集群实际作战运用的重要一环。构建以动态一致性联盟算法(Dynamic Consensus-based Grouping Algorithm,DCBGA)为核心的无人机集群决策控制流程框架。为无人机集群设计通信受限条件的通信组网模型,并引入动态自适应机制,以有效应对高动态任务场景;基于“作战环”理论,对网络架构下的异构单元非线性作战效能聚合效果进行描述,并纳入全局效益函数,牵引异构无人机协同作战联盟组建;将联盟组建过程划分为目标选择、一致性和信息与状态更新三个阶段,采用动态一致性联盟算法支撑无人机集群自下而上完成各阶段的信息决策控制。仿真结果表明,新构建的算法体系可以有效推动异构无人机集群实现作战联盟组建,完成协同打击作战任务,具有较好的动态适应性及规模扩展性,在对抗环境下展现出良好的韧性。

航母编队应对电子战飞机战法研究

论文以美军EA-18G电子战飞机为例,研究航母编队应对电子战飞机的作战方法。首先,分析电子战飞机作战能力、主要战法、强弱点及对航母编队的影响;随后,根据电子战飞机的主要特点及作战能力,从预警探测、拦截防抗、对抗反击、应对掩护导弹等方面研提出航母编队作战方法;最后对未来舰艇雷达电子设备发展提出建议。

论新质公安战斗力的形成条件与实现路径

新时代,随着社会经济的快速发展和信息技术的广泛应用,公安机关面临着日益复杂的治安形势和越来越多的挑战。新质公安战斗力的构建已成为推进国家治理体系和治理能力现代化的重要环节。为了更好地维护国家安全和社会稳定,提升新质公安战斗力成为迫切需要。本文从新质公安战斗力的内涵与特征出发,分析新质公安战斗力的形成条件,探讨新质公安战斗力的实现路径,为公安工作现代化提供理论支撑。

UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning

In order to improve the autonomous ability of unmanned aerial vehicles(UAV)to implement air combat mission,many artificial intelligence-based autonomous air combat maneuver decision-making studies have been carried out,but these studies are often aimed at individual decision-making in 1 v1 scenarios which rarely happen in actual air combat.Based on the research of the 1 v1 autonomous air combat maneuver decision,this paper builds a multi-UAV cooperative air combat maneuver decision model based on multi-agent reinforcement learning.Firstly,a bidirectional recurrent neural network(BRNN)is used to achieve communication between UAV individuals,and the multi-UAV cooperative air combat maneuver decision model under the actor-critic architecture is established.Secondly,through combining with target allocation and air combat situation assessment,the tactical goal of the formation is merged with the reinforcement learning goal of every UAV,and a cooperative tactical maneuver policy is generated.The simulation results prove that the multi-UAV cooperative air combat maneuver decision model established in this paper can obtain the cooperative maneuver policy through reinforcement learning,the cooperative maneuver policy can guide UAVs to obtain the overall situational advantage and defeat the opponents under tactical cooperation.

A Multi-UCAV cooperative occupation method based on weapon engagement zones for beyond-visual-range air combat

Recent advances in on-board radar and missile capabilities,combined with individual payload limitations,have led to increased interest in the use of unmanned combat aerial vehicles(UCAVs)for cooperative occupation during beyond-visual-range(BVR)air combat.However,prior research on occupational decision-making in BVR air combat has mostly been limited to one-on-one scenarios.As such,this study presents a practical cooperative occupation decision-making methodology for use with multiple UCAVs.The weapon engagement zone(WEZ)and combat geometry were first used to develop an advantage function for situational assessment of one-on-one engagement.An encircling advantage function was then designed to represent the cooperation of UCAVs,thereby establishing a cooperative occupation model.The corresponding objective function was derived from the one-on-one engagement advantage function and the encircling advantage function.The resulting model exhibited similarities to a mixed-integer nonlinear programming(MINLP)problem.As such,an improved discrete particle swarm optimization(DPSO)algorithm was used to identify a solution.The occupation process was then converted into a formation switching task as part of the cooperative occupation model.A series of simulations were conducted to verify occupational solutions in varying situations,including two-on-two engagement.Simulated results showed these solutions varied with initial conditions and weighting coefficients.This occupation process,based on formation switching,effectively demonstrates the viability of the proposed technique.These cooperative occupation results could provide a theoretical framework for subsequent research in cooperative BVR air combat.

Cooperative Decision-Making for Multiple UAVs Autonomous Confrontation

This paper presents a rule-based framework for addressing decision-making problems within the context of the\UI-STRIVE"Competition.First,two distinct autonomous confrontation scenarios are described:autonomous air combat and cooperative interception.Second,a State-Event-Condition-Action(SECA)decision-making framework is developed,which integrates thefinite state machine and event-condition-action frameworks.This framework provides three products to describe rules,i.e.the SECA model,the SECA state chart,and the SECA rule description.Third,the situation assessment and target assignment during autonomous air combat are investigated,and the mathematical models are established.Finally,the decisionmaking model's rationality and feasibility are verified through data simulation and analysis.

面向军事群体的聚合及解聚可视化控制模型

作战编队是军事领域特有的军事群体组织方式,具有广泛的军事应用,针对战场态势中作战编队的聚合/解聚可视化问题,基于知识图谱及模型-视图-控制器(MVC)设计模式,提出了面向作战编队的聚合/解聚可视化控制模型,主要研究作战编队基于MVC的架构设计、作战编队基于知识图谱语义建模、基于主成分分析(PCA)的作战编队区域几何辅助对象构建、作战编队解聚/聚合ADLOD显示及地图比例尺控制以及作战编队聚合/解聚可视化模型的向量形式表征,该模型既弥补了军事标绘相关标准中作战群体方面研究的空白,也为联合作战态势多分辨率显示优化提供了基于作战编队聚合简化的新技术途径,同时,提供的可视化手段可加深对战场综合态势的认知,提升作战指挥决策水平,军事应用前景良好。

有人/无人机协同系统及关键技术综述

有人/无人机协同系统是在无人机系统基础上发展起来的一种新型协同作战系统.随着任务复杂度的不断加深,仅凭无人机不能满足各类任务需求,有人机承担的指挥、控制和决策功能的重要性逐步增强,是系统任务执行效能得以提升的关键要素.凭借无人机强大的感知、计算、通讯能力以及机载飞行员的高级智慧和经验,有人机和无人机可以实现协同编队,完成各种复杂任务.深入分析了有人/无人机协同系统的架构及组成,总结了目前的发展状况,提炼和归纳了系统的关键技术.最后对系统未来的发展方向进行了展望.

非完备信息下的超视距空战双机协同战术识别

针对超视距(BVR)空战过程中,受探测装置性能限制和敌方干扰等原因,导致目标信息易缺失,从而难以实时准确地识别敌方协同空战战术的问题,提出了一种基于动态贝叶斯网络(DBN)与参数学习的超视距空战双机协同战术识别方法。分析了超视距空战条件下的双机协同战术特征,根据长机和僚机的职能分工、当前态势及机动动作,构建了识别网络模型;为提高模型对双机协同战术的识别概率,采用期望最大参数学习方法优化网络参数;基于自回归模型对缺失目标信息进行修补,提出非完备信息下的双机协同战术识别推理算法。通过开展空战对抗仿真实验,验证了双机协同战术识别方法对于非完备信息下的超视距空战双机协同战术具有较高的识别概率和较好的实时性。

基于Lanchester方程的编队舰空导弹作战能力评估模型

针对编队舰空导弹作战能力对抗评估问题,在构建舰空导弹初始作战能力指数、双方损耗效率系数、正常发挥因子、指挥自动化系数评估模型基础上,构建了基于Lanchester方程的单舰舰空导弹作战能力评估模型,进而分析编队对抗损耗效率系数及武器分配系数评估模型,构建了基于Lanchester方程的编队舰空导弹协同作战能力评估模型,评定编队舰空导弹作战能力,为编队舰空导弹协同作战能力评估提供模型依据。

基于约束跟随的无人战车编队控制

针对无人战车编队系统控制过程中需要克服非线性和不确定性的问题,提出一种基于约束跟随的自适应鲁棒控制方法。以伺服约束的形式描述被控系统需要遵循的约束,并建立约束跟随误差,将编队控制问题转化为一类近似约束跟随问题。考虑编队执行任务时需要到达指定的作战区域,在行驶约束的基础上添加到达约束;设计自适应鲁棒控制器,利用Lyapunov稳定性分析方法,证明系统误差一致有界和一致最终有界。进而保证编队系统在复杂的不确定因素干扰下,仍具有较好的被控精度和系统稳定性。仿真实验结果表明:该自适应鲁棒控制器较好地解决系统中的不确定性,在满足编队系统队形要求的同时,驱使战车编队到达目标区域。