Mastering air combat game with deep reinforcement learning

Reinforcement learning has been applied to air combat problems in recent years,and the idea of curriculum learning is often used for reinforcement learning,but traditional curriculum learning suffers from the problem of plasticity loss in neural networks.Plasticity loss is the difficulty of learning new knowledge after the network has converged.To this end,we propose a motivational curriculum learning distributed proximal policy optimization(MCLDPPO)algorithm,through which trained agents can significantly outperform the predictive game tree and mainstream reinforcement learning methods.The motivational curriculum learning is designed to help the agent gradually improve its combat ability by observing the agent's unsatisfactory performance and providing appropriate rewards as a guide.Furthermore,a complete tactical maneuver is encapsulated based on the existing air combat knowledge,and through the flexible use of these maneuvers,some tactics beyond human knowledge can be realized.In addition,we designed an interruption mechanism for the agent to increase the frequency of decisionmaking when the agent faces an emergency.When the number of threats received by the agent changes,the current action is interrupted in order to reacquire observations and make decisions again.Using the interruption mechanism can significantly improve the performance of the agent.To simulate actual air combat better,we use digital twin technology to simulate real air battles and propose a parallel battlefield mechanism that can run multiple simulation environments simultaneously,effectively improving data throughput.The experimental results demonstrate that the agent can fully utilize the situational information to make reasonable decisions and provide tactical adaptation in the air combat,verifying the effectiveness of the algorithmic framework proposed in this paper.

Optimal confrontation position selecting games model and its application to one-on-one air combat

In the air combat process,confrontation position is the critical factor to determine the confrontation situation,attack effect and escape probability of UAVs.Therefore,selecting the optimal confrontation position becomes the primary goal of maneuver decision-making.By taking the position as the UAV’s maneuver strategy,this paper constructs the optimal confrontation position selecting games(OCPSGs)model.In the OCPSGs model,the payoff function of each UAV is defined by the difference between the comprehensive advantages of both sides,and the strategy space of each UAV at every step is defined by its accessible space determined by the maneuverability.Then we design the limit approximation of mixed strategy Nash equilibrium(LAMSNQ)algorithm,which provides a method to determine the optimal probability distribution of positions in the strategy space.In the simulation phase,we assume the motions on three directions are independent and the strategy space is a cuboid to simplify the model.Several simulations are performed to verify the feasibility,effectiveness and stability of the algorithm.

A function-based behavioral modeling method for air combat simulation

Today’s air combat has reached a high level of uncertainty where continuous or discrete variables with crisp values cannot be properly represented using fuzzy sets. With a set of membership functions, fuzzy logic is well-suited to tackle such complex states and actions. However, it is not necessary to fuzzify the variables that have definite discrete semantics.Hence, the aim of this study is to improve the level of model abstraction by proposing multiple levels of cascaded hierarchical structures from the perspective of function, namely, the functional decision tree. This method is developed to represent behavioral modeling of air combat systems, and its metamodel,execution mechanism, and code generation can provide a sound basis for function-based behavioral modeling. As a proof of concept, an air combat simulation is developed to validate this method and the results show that the fighter Alpha built using the proposed framework provides better performance than that using default scripts.

Cooperative maneuver decision making for multi-UAV air combat based on incomplete information dynamic game

Cooperative autonomous air combat of multiple unmanned aerial vehicles(UAVs)is one of the main combat modes in future air warfare,which becomes even more complicated with highly changeable situation and uncertain information of the opponents.As such,this paper presents a cooperative decision-making method based on incomplete information dynamic game to generate maneuver strategies for multiple UAVs in air combat.Firstly,a cooperative situation assessment model is presented to measure the overall combat situation.Secondly,an incomplete information dynamic game model is proposed to model the dynamic process of air combat,and a dynamic Bayesian network is designed to infer the tactical intention of the opponent.Then a reinforcement learning framework based on multiagent deep deterministic policy gradient is established to obtain the perfect Bayes-Nash equilibrium solution of the air combat game model.Finally,a series of simulations are conducted to verify the effectiveness of the proposed method,and the simulation results show effective synergies and cooperative tactics.

Autonomous air combat decision-making of UAV based on parallel self-play reinforcement learning

Aiming at addressing the problem of manoeuvring decision-making in UAV air combat,this study establishes a one-to-one air combat model,defines missile attack areas,and uses the non-deterministic policy Soft-Actor-Critic(SAC)algorithm in deep reinforcement learning to construct a decision model to realize the manoeuvring process.At the same time,the complexity of the proposed algorithm is calculated,and the stability of the closed-loop system of air combat decision-making controlled by neural network is analysed by the Lyapunov function.This study defines the UAV air combat process as a gaming process and proposes a Parallel Self-Play training SAC algorithm(PSP-SAC)to improve the generalisation performance of UAV control decisions.Simulation results have shown that the proposed algorithm can realize sample sharing and policy sharing in multiple combat environments and can significantly improve the generalisation ability of the model compared to independent training.

HEURISTIC PARTICLE SWARM OPTIMIZATION ALGORITHM FOR AIR COMBAT DECISION-MAKING ON CMTA

Combining the heuristic algorithm （HA） developed based on the specific knowledge of the cooperative multiple target attack （CMTA） tactics and the particle swarm optimization （PSO）, a heuristic particle swarm optimization （HPSO） algorithm is proposed to solve the decision-making （DM） problem. HA facilitates to search the local optimum in the neighborhood of a solution, while the PSO algorithm tends to explore the search space for possible solutions. Combining the advantages of HA and PSO, HPSO algorithms can find out the global optimum quickly and efficiently. It obtains the DM solution by seeking for the optimal assignment of missiles of friendly fighter aircrafts （FAs） to hostile FAs. Simulation results show that the proposed algorithm is superior to the general PSO algorithm and two GA based algorithms in searching for the best solution to the DM problem.

Autonomous air combat maneuver decision using Bayesian inference and moving horizon optimization

To reach a higher level of autonomy for unmanned combat aerial vehicle(UCAV) in air combat games, this paper builds an autonomous maneuver decision system. In this system,the air combat game is regarded as a Markov process, so that the air combat situation can be effectively calculated via Bayesian inference theory. According to the situation assessment result,adaptively adjusts the weights of maneuver decision factors, which makes the objective function more reasonable and ensures the superiority situation for UCAV. As the air combat game is characterized by highly dynamic and a significant amount of uncertainty,to enhance the robustness and effectiveness of maneuver decision results, fuzzy logic is used to build the functions of four maneuver decision factors. Accuracy prediction of opponent aircraft is also essential to ensure making a good decision; therefore, a prediction model of opponent aircraft is designed based on the elementary maneuver method. Finally, the moving horizon optimization strategy is used to effectively model the whole air combat maneuver decision process. Various simulations are performed on typical scenario test and close-in dogfight, the results sufficiently demonstrate the superiority of the designed maneuver decision method.

UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning

In order to improve the autonomous ability of unmanned aerial vehicles(UAV)to implement air combat mission,many artificial intelligence-based autonomous air combat maneuver decision-making studies have been carried out,but these studies are often aimed at individual decision-making in 1 v1 scenarios which rarely happen in actual air combat.Based on the research of the 1 v1 autonomous air combat maneuver decision,this paper builds a multi-UAV cooperative air combat maneuver decision model based on multi-agent reinforcement learning.Firstly,a bidirectional recurrent neural network(BRNN)is used to achieve communication between UAV individuals,and the multi-UAV cooperative air combat maneuver decision model under the actor-critic architecture is established.Secondly,through combining with target allocation and air combat situation assessment,the tactical goal of the formation is merged with the reinforcement learning goal of every UAV,and a cooperative tactical maneuver policy is generated.The simulation results prove that the multi-UAV cooperative air combat maneuver decision model established in this paper can obtain the cooperative maneuver policy through reinforcement learning,the cooperative maneuver policy can guide UAVs to obtain the overall situational advantage and defeat the opponents under tactical cooperation.

Situation assessment for air combat based on novel semi-supervised naive Bayes

A method is proposed to resolve the typical problem of air combat situation assessment. Taking the one-to-one air combat as an example and on the basis of air combat data recorded by the air combat maneuvering instrument, the problem of air combat situation assessment is equivalent to the situation classification problem of air combat data. The fuzzy C-means clustering algorithm is proposed to cluster the selected air combat sample data and the situation classification of the data is determined by the data correlation analysis in combination with the clustering results and the pilots＇ description of the air combat process. On the basis of semi-supervised naive Bayes classifier, an improved algorithm is proposed based on data classification confidence, through which the situation classification of air combat data is carried out. The simulation results show that the improved algorithm can assess the air combat situation effectively and the improvement of the algorithm can promote the classification performance without significantly affecting the efficiency of the classifier.

Air combat decision-making of multiple UCAVs based on constraint strategy games

Game theory can be applied to the air combat decision-making problem of multiple unmanned combat air vehicles(UCAVs).However,it is difficult to have satisfactory decision-making results completely relying on air combat situation information,because there is a lot of time-sensitive information in a complex air combat environment.In this paper,a constraint strategy game approach is developed to generate intelligent decision-making for multiple UCAVs in complex air combat environment with air combat situation information and time-sensitive information.Initially,a constraint strategy game is employed to model attack-defense decision-making problem in complex air combat environment.Then,an algorithm is proposed for solving the constraint strategy game based on linear programming and linear inequality(CSG-LL).Finally,an example is given to illustrate the effectiveness of the proposed approach.

Extracting Decision Rules for Cooperative Team Air Combat Based on Rough Set Theory

In order to reduce redundant features in air combat information and to meet the requirements of real-time decision in combat, rough set theory is introduced to the tactical decision analysis in cooperative team air combat. An algorithm of attribute reduction for extracting key combat information and generating tactical rules from given air combat databases is presented. Then, considering the practical requirements of team combat, a method for reduction of attribute-values under single decision attribute is extended to the reduction under multi-decision attributes. Finally, the algorithm is verified with an example for tactical choices in team air combat. The results show that, the redundant attributes in air combat information can be reduced, and that the main combat attributes, i.e., the information about radar command and medium-range guided missile, can be obtained with the algorithm mentioned above, moreover, the minimal reduced strategy for tactical decision can be generated without losing the result of key information classification. The decision rules extracted agree with the real situation of team air combat.

Sequential maneuvering decisions based on multi-stage influence diagram in air combat

A multi-stage influence diagram is used to model the pilot＇s sequential decision making in one on one air combat. The model based on the multi-stage influence diagram graphically describes the elements of decision process, and contains a point-mass model for the dynamics of an aircraft and takes into account the decision maker＇s preferences under uncertain conditions. Considering an active opponent, the opponent＇s maneuvers can be modeled stochastically. The solution of multistage influence diagram can be obtained by converting the multistage influence diagram into a two-level optimization problem. The simulation results show the model is effective.

Approach to WTA in air combat using IAFSA-IHS algorithm

In this paper, a static weapon target assignment（WTA）problem is studied. As a critical problem in cooperative air combat,outcome of WTA directly influences the battle. Along with the cost of weapons rising rapidly, it is indispensable to design a target assignment model that can ensure minimizing targets survivability and weapons consumption simultaneously. Afterwards an algorithm named as improved artificial fish swarm algorithm-improved harmony search algorithm（IAFSA-IHS） is proposed to solve the problem. The effect of the proposed algorithm is demonstrated in numerical simulations, and results show that it performs positively in searching the optimal solution and solving the WTA problem.

Task assignment under constraint of timing sequential for cooperative air combat

According to the previous achievement, the task assignment under the constraint of timing continuity for a cooperative air combat is studied. An extensive task assignment scenario with the background of the cooperative air combat is proposed. The utility and time of executing a task as well as the continuous combat ability are defined. The concept of the matching method of weapon and target is modified based on the analysis of the air combat scenario. The constraint framework is also redefined according to a new objective function. The constraints of timing and continuity are formulated with a new method, at the same time, the task assignment and integer programming models of the cooperative combat are established. Finally, the assignment problem is solved using the integrated linear programming software and the simulation shows that it is feasible to apply this modified model in the cooperative air combat for tasks cooperation and it is also efficient to optimize the resource assignment.

A Multi-UCAV cooperative occupation method based on weapon engagement zones for beyond-visual-range air combat

Recent advances in on-board radar and missile capabilities,combined with individual payload limitations,have led to increased interest in the use of unmanned combat aerial vehicles(UCAVs)for cooperative occupation during beyond-visual-range(BVR)air combat.However,prior research on occupational decision-making in BVR air combat has mostly been limited to one-on-one scenarios.As such,this study presents a practical cooperative occupation decision-making methodology for use with multiple UCAVs.The weapon engagement zone(WEZ)and combat geometry were first used to develop an advantage function for situational assessment of one-on-one engagement.An encircling advantage function was then designed to represent the cooperation of UCAVs,thereby establishing a cooperative occupation model.The corresponding objective function was derived from the one-on-one engagement advantage function and the encircling advantage function.The resulting model exhibited similarities to a mixed-integer nonlinear programming(MINLP)problem.As such,an improved discrete particle swarm optimization(DPSO)algorithm was used to identify a solution.The occupation process was then converted into a formation switching task as part of the cooperative occupation model.A series of simulations were conducted to verify occupational solutions in varying situations,including two-on-two engagement.Simulated results showed these solutions varied with initial conditions and weighting coefficients.This occupation process,based on formation switching,effectively demonstrates the viability of the proposed technique.These cooperative occupation results could provide a theoretical framework for subsequent research in cooperative BVR air combat.

Online hierarchical recognition method for target tactical intention in beyond-visual-range air combat

Online accurate recognition of target tactical intention in beyond-visual-range (BVR) air combat is an important basis for deep situational awareness and autonomous air combat decision-making, which can create pre-emptive tactical opportunities for the fighter to gain air superiority. The existing methods to solve this problem have some defects such as dependence on empirical knowledge, difficulty in interpreting the recognition results, and inability to meet the requirements of actual air combat. So an online hierarchical recognition method for target tactical intention in BVR air combat based on cascaded support vector machine (CSVM) is proposed in this study. Through the mechanism analysis of BVR air combat, the instantaneous and cumulative feature information of target trajectory and relative situation information are introduced successively using online automatic decomposition of target trajectory and hierarchical progression. Then the hierarchical recognition model from target maneuver element, tactical maneuver to tactical intention is constructed. The CSVM algorithm is designed for solving this model, and the computational complexity is decomposed by the cascaded structure to overcome the problems of convergence and timeliness when the dimensions and number of training samples are large. Meanwhile, the recognition result of each layer can be used to support the composition analysis and interpretation of target tactical intention. The simulation results show that the proposed method can effectively realize multi-dimensional online accurate recognition of target tactical intention in BVR air combat.

Weapon configuration, allocation and route planning with time windows for multiple unmanned combat air vehicles

Unmanned combat air vehicles(UCAVs) mission planning is a fairly complicated global optimum problem. Military attack missions often employ a fleet of UCAVs equipped with weapons to attack a set of known targets. A UCAV can carry different weapons to accomplish different combat missions. Choice of different weapons will have different effects on the final combat effectiveness. This work presents a mixed integer programming model for simultaneous weapon configuration and route planning of UCAVs, which solves the problem optimally using the IBM ILOG CPLEX optimizer for simple missions. This paper develops a heuristic algorithm to handle the medium-scale and large-scale problems. The experiments demonstrate the performance of the heuristic algorithm in solving the medium scale and large scale problems. Moreover, we give suggestions on how to select the most appropriate algorithm to solve different scale problems.

Service-oriented air combat simulation architecture

Service-oriented air combat simulation architecture is proposed.The core design goal is high agility which represents the ability to accommodate the simulation requirements change.Its main idea is to design model units as services that can communicate and interoperate with any other services at runtime.A service is autonomous and is fully defined by a description contract which contains some combination of syntactic,semantic,and behavioral information.Based on the architecture,air combat simulation system can be described as an abstract composition of description contracts.It becomes concrete at run time as services that implement the constituent description contracts are discovered and bind.The whole process is a continuous run-time activity that responds to simulation needs and the availability of services.This provides benefits of implementation transparency and minimal dependency between models.Thus,simulation system can minimize the impact of change on it and increase the overall efficiency to respond to requirements change.

无人机集群对抗决策算法研究综述

无人机集群博弈对抗已经成为未来战争的发展趋势,无人机对抗决策算法的选择对提升无人机集群作战能力至关重要。本文深入探讨了基于规则的、基于博弈论的和基于神经网络的三大类无人机集群博弈对抗决策算法,并对它们的优势和局限性进行了全面分析与总结。在此基础上,提出将“基于多智能体强化学习的信用分配模型”和“基于角色的多智能体强化学习模型”应用于无人机集群博弈对抗的研究思路。最后,强调了选择适当的决策算法对于提高无人机集群作战效能的重要性,并为未来无人机对抗决策的发展提出了有益的建议,为相关领域的研究和应用提供了深入见解。

智能化空天防御作战体系架构设计方法研究

在潜在威胁日趋智能化和当前智能技术不断进步的双重推动下,空天防御作战体系不是装备和技术的线性叠加与机械堆砌,而是一个各作战要素有机融合、动态适变与持续演进的综合智能体。在分析智能化空天防御作战体系特征的基础上,创新智能化设计范式与理念,提出一种基于智能体网络的体系架构设计方法,从兵力节点、兵力关系、适变机制3个方面探讨了智能化赋能架构设计的关键技术。该方法为空天防御力量的顶层设计提供了智能化方法参考,有助于提升空天防御作战体系建设与运用的智能化、网络化水平。