UAV maneuvering decision-making algorithm based on deep reinforcement learning under the guidance of expert experience

Autonomous umanned aerial vehicle(UAV) manipulation is necessary for the defense department to execute tactical missions given by commanders in the future unmanned battlefield. A large amount of research has been devoted to improving the autonomous decision-making ability of UAV in an interactive environment, where finding the optimal maneuvering decisionmaking policy became one of the key issues for enabling the intelligence of UAV. In this paper, we propose a maneuvering decision-making algorithm for autonomous air-delivery based on deep reinforcement learning under the guidance of expert experience. Specifically, we refine the guidance towards area and guidance towards specific point tasks for the air-delivery process based on the traditional air-to-surface fire control methods.Moreover, we construct the UAV maneuvering decision-making model based on Markov decision processes(MDPs). Specifically, we present a reward shaping method for the guidance towards area and guidance towards specific point tasks using potential-based function and expert-guided advice. The proposed algorithm could accelerate the convergence of the maneuvering decision-making policy and increase the stability of the policy in terms of the output during the later stage of training process. The effectiveness of the proposed maneuvering decision-making policy is illustrated by the curves of training parameters and extensive experimental results for testing the trained policy.

Multiple model PHD filter for tracking sharply maneuvering targets using recursive RANSAC based adaptive birth estimation

An algorithm to track multiple sharply maneuvering targets without prior knowledge about new target birth is proposed. These targets are capable of achieving sharp maneuvers within a short period of time, such as drones and agile missiles.The probability hypothesis density (PHD) filter, which propagates only the first-order statistical moment of the full target posterior, has been shown to be a computationally efficient solution to multitarget tracking problems. However, the standard PHD filter operates on the single dynamic model and requires prior information about target birth distribution, which leads to many limitations in terms of practical applications. In this paper,we introduce a nonzero mean, white noise turn rate dynamic model and generalize jump Markov systems to multitarget case to accommodate sharply maneuvering dynamics. Moreover, to adaptively estimate newborn targets’information, a measurement-driven method based on the recursive random sampling consensus (RANSAC) algorithm is proposed. Simulation results demonstrate that the proposed method achieves significant improvement in tracking multiple sharply maneuvering targets with adaptive birth estimation.

Cooperative guidance law based on super-twisting observer for target maneuvering

To solve the problem that multiple missiles should simultaneously attack unmeasurable maneuvering targets,a guidance law with temporal consistency constraint based on the super-twisting observer is proposed.Firstly,the relative motion equations between multiple missiles and targets are established,and the topological model among multiple agents is considered.Secondly,based on the temporal consistency constraint,a cooperative guidance law for simultaneous arrival with finite-time convergence is derived.Finally,the unknown target maneuver-ing is regarded as bounded interference.Based on the second-order sliding mode theory,a super-twisting sliding mode observer is devised to observe and track the bounded interfer-ence,and the stability of the observer is proved.Compared with the existing research,this approach only needs to obtain the sliding mode variable which simplifies the design process.The simulation results show that the designed cooperative guidance law for maneuvering targets achieves the expected effect.It ensures successful cooperative attacks,even when confronted with strong maneuvering targets.

Situational continuity-based air combat autonomous maneuvering decision-making

In order to improve the performance of UAV's autonomous maneuvering decision-making,this paper proposes a decision-making method based on situational continuity.The algorithm in this paper designs a situation evaluation function with strong guidance,then trains the Long Short-Term Memory(LSTM)under the framework of Deep Q Network(DQN)for air combat maneuvering decision-making.Considering the continuity between adjacent situations,the method takes multiple consecutive situations as one input of the neural network.To reflect the difference between adjacent situations,the method takes the difference of situation evaluation value as the reward of reinforcement learning.In different scenarios,the algorithm proposed in this paper is compared with the algorithm based on the Fully Neural Network(FNN)and the algorithm based on statistical principles respectively.The results show that,compared with the FNN algorithm,the algorithm proposed in this paper is more accurate and forwardlooking.Compared with the algorithm based on the statistical principles,the decision-making of the algorithm proposed in this paper is more efficient and its real-time performance is better.

Optimal maneuvering strategy of spacecraft evasion based on angles-only measurement and observability analysis

Spacecraft orbit evasion is an effective method to ensure space safety. In the spacecraft’s orbital plane, the space non-cooperate target with autonomous approaching to the spacecraft may have a dangerous rendezvous. To deal with this problem, an optimal maneuvering strategy based on the relative navigation observability degree is proposed with angles-only measurements. A maneuver evasion relative navigation model in the spacecraft’s orbital plane is constructed and the observability measurement criteria with process noise and measurement noise are defined based on the posterior Cramer-Rao lower bound. Further, the optimal maneuver evasion strategy in spacecraft’s orbital plane based on the observability is proposed. The strategy provides a new idea for spacecraft to evade safety threats autonomously. Compared with the spacecraft evasion problem based on the absolute navigation, more accurate evasion results can be obtained. The simulation indicates that this optimal strategy can weaken the system’s observability and reduce the state estimation accuracy of the non-cooperative target, making it impossible for the non-cooperative target to accurately approach the spacecraft.

An Effective Multiple Model Least Squares Method in Tracking of a Maneuvering Target

A polynomial model, time origin shifting model(TOSM, is used to describe the trajectory of a moving target .Based on TOSM, a recursive laeast squares(RLS) algorithm with varied forgetting factor is derived for tracking of a non-maneuvering target. In order to apply this algorithm to maneuvering targets tracking ,a tracking signal is performed on-line to determine what kind of TOSm will be in effect to track a target with different dynamics. An effective multiple model least squares filtering and forecasting method dadpted to real tracking of a maneuvering target is formulated. The algorithm is computationally more effcient than Kalman filter and the percentage improvement from simulations show both of them are considerably alike to some extent.

Maneuvering target tracking using threshold interacting multiple model algorithm

To avoid missing track caused by the target maneuvers in automatic target tracking system, a new maneuvering target tracking technique called threshold interacting multiple model （TIMM） is proposed. This algorithm is based on the interacting multiple model （IMM） method and applies a threshold controller to improve tracking accuracy. It is also applicable to other advanced algorithms of IMM. In this research, we also compare the position and velocity root mean square （RMS） errors of TIMM and IMM algorithms with two different examples. Simulation results show that the TIMM algorithm is superior to the traditional IMM alzorithm in estimation accuracy.

CFD Investigations of Ship Maneuvering in Waves Using naoe-FOAM-SJTU Solver

Ship maneuvering in waves includes the performance of ship resistance, seakeeping, propulsion, and maneuverability. It is a complex hydrodynamic problem with the interaction of many factors. With the purpose of directly predicting the behavior of ship maneuvering in waves, a CFD solver named naoe-FOAM-SJTU is developed by the Computational Marine Hydrodynamics Lab(CMHL) in Shanghai Jiao Tong University. The solver is based on open source platform OpenFOAM and has introduced dynamic overset grid technology to handle complex ship hull-propeller-rudder motion system. Maneuvering control module based on feedback control mechanism is also developed to accurately simulate corresponding motion behavior of free running ship maneuver. Inlet boundary wavemaker and relaxation zone technique is used to generate desired waves. Based on the developed modules, unsteady Reynolds-averaged Navier-Stokes(RANS) computations are carried out for several validation cases of free running ship maneuver in waves including zigzag, turning circle, and course keeping maneuvers. The simulation results are compared with available benchmark data. Ship motions, trajectories, and other maneuvering parameters are consistent with available experimental data, which indicate that the present solver can be suitable and reliable in predicting the performance of ship maneuvering in waves. Flow visualizations, such as free surface elevation, wake flow, vortical structures, are presented to explain the hydrodynamic performance of ship maneuvering in waves. Large flow separation can be observed around propellers and rudders. It is concluded that RANS approach is not accurate enough for predicting ship maneuvering in waves with large flow separations and detached eddy simulation(DES) or large eddy simulation(LES) computations are required to improve the prediction accuracy.

Modeling of Ship Maneuvering Motion Using Neural Networks

In this paper, Neural Networks （NNs） are used in the modeling of ship maneuvering motion. A nonlinear response model and a linear hydrodynamic model of ship maneuvering motion are also investigated. The maneuverability indices and linear non-dimensional hydrodynamic derivatives in the models are identified by using two-layer feed forward NNs. The stability of parametric estimation is confirmed. Then, the ship maneuvering motion is predicted based on the obtained models. A comparison between the predicted results and the model test results demonstrates the validity of the proposed modeling method.

Maneuvering target tracking of UAV based on MN-DDPG and transfer learning

Tracking maneuvering target in real time autonomously and accurately in an uncertain environment is one of the challenging missions for unmanned aerial vehicles(UAVs).In this paper,aiming to address the control problem of maneuvering target tracking and obstacle avoidance,an online path planning approach for UAV is developed based on deep reinforcement learning.Through end-to-end learning powered by neural networks,the proposed approach can achieve the perception of the environment and continuous motion output control.This proposed approach includes:(1)A deep deterministic policy gradient(DDPG)-based control framework to provide learning and autonomous decision-making capability for UAVs;(2)An improved method named MN-DDPG for introducing a type of mixed noises to assist UAV with exploring stochastic strategies for online optimal planning;and(3)An algorithm of taskdecomposition and pre-training for efficient transfer learning to improve the generalization capability of UAV’s control model built based on MN-DDPG.The experimental simulation results have verified that the proposed approach can achieve good self-adaptive adjustment of UAV’s flight attitude in the tasks of maneuvering target tracking with a significant improvement in generalization capability and training efficiency of UAV tracking controller in uncertain environments.

An impact angle constraint integral sliding mode guidance law for maneuvering targets interception

An integral sliding mode guidance law(ISMGL)combined with the advantages of the integral sliding mode control(SMC)method is designed to address maneuvering target interception problems with impact angle constraints.The relative motion equation of the missile and the target considering the impact angle constraint is established in the longitudinal plane,and an integral sliding mode surface is constructed.The proposed guidance law resolves the existence of a steady-state error problem in the traditional SMC.Such a guidance law ensures that the missile hits the target with an ideal impact angle in finite time and the missile is kept highly robust throughout the interception process.By adopting the dynamic surface control method,the ISMGL is designed considering the impact angle constraints and the autopilot dynamic characteristics.According to the Lyapunov stability theorem,all states of the closed-loop system are finally proven to be uniformly bounded.Simulation results are compared with the general sliding mode guidance law and the trajectory shaping guidance law,and the findings verify the effectiveness and superiority of the ISMGL.

Modified unscented Kalman filter using modified filter gain and variance scale factor for highly maneuvering target tracking

To improve the low tracking precision caused by lagged filter gain or imprecise state noise when the target highly maneuvers, a modified unscented Kalman filter algorithm based on the improved filter gain and adaptive scale factor of state noise is presented. In every filter process, the estimated scale factor is used to update the state noise covariance Qk, and the improved filter gain is obtained in the filter process of unscented Kalman filter （UKF） via predicted variance Pk｜k-1, which is similar to the standard Kalman filter. Simulation results show that the proposed algorithm provides better accuracy and ability to adapt to the highly maneuvering target compared with the standard UKF.

Adaptive Maneuvering Frequency Method of Current Statistical Model

Current statistical model(CSM) has a good performance in maneuvering target tracking. However, the fixed maneuvering frequency will deteriorate the tracking results, such as a serious dynamic delay, a slowly converging speedy and a limited precision when using Kalman filter(KF) algorithm. In this study, a new current statistical model and a new Kalman filter are proposed to improve the performance of maneuvering target tracking. The new model which employs innovation dominated subjection function to adaptively adjust maneuvering frequency has a better performance in step maneuvering target tracking, while a fluctuant phenomenon appears. As far as this problem is concerned, a new adaptive fading Kalman filter is proposed as well. In the new Kalman filter, the prediction values are amended in time by setting judgment and amendment rules,so that tracking precision and fluctuant phenomenon of the new current statistical model are improved. The results of simulation indicate the effectiveness of the new algorithm and the practical guiding significance.

Sequential maneuvering decisions based on multi-stage influence diagram in air combat

A multi-stage influence diagram is used to model the pilot＇s sequential decision making in one on one air combat. The model based on the multi-stage influence diagram graphically describes the elements of decision process, and contains a point-mass model for the dynamics of an aircraft and takes into account the decision maker＇s preferences under uncertain conditions. Considering an active opponent, the opponent＇s maneuvers can be modeled stochastically. The solution of multistage influence diagram can be obtained by converting the multistage influence diagram into a two-level optimization problem. The simulation results show the model is effective.

Sensor scheduling for ground maneuvering target tracking in presence of detection blind zone

Continuous and stable tracking of the ground maneuvering target is a challenging problem due to the complex terrain and high clutter. A collaborative tracking method of the multisensor network is presented for the ground maneuvering target in the presence of the detection blind zone(DBZ). First, the sensor scheduling process is modeled within the partially observable Markov decision process(POMDP) framework. To evaluate the target tracking accuracy of the sensor, the Fisher information is applied to constructing the reward function. The key of the proposed scheduling method is forecasting and early decisionmaking. Thus, an approximate method based on unscented sampling is presented to estimate the target state and the multi-step scheduling reward over the prediction time horizon. Moreover, the problem is converted into a nonlinear optimization problem, and a fast search algorithm is given to solve the sensor scheduling scheme quickly. Simulation results demonstrate the proposed nonmyopic scheduling method(Non-MSM) has a better target tracking accuracy compared with traditional methods.

Multiple model efficient particle filter based track-before-detect for maneuvering weak targets

It is a tough problem to jointly detect and track a weak target, and it becomes even more challenging when the target is maneuvering. The above problem is formulated by using the Bayesian theory and a multiple model(MM) based filter is proposed. The filter presented uses the MM method to accommodate the multiple motions that a maneuvering target may travel under by adding a random variable representing the motion model to the target state. To strengthen the efficiency performance of the filter,the target existence variable is separated from the target state and the existence probability is calculated in a more efficient way. To examine the performance of the MM based approach, a typical track-before-detect(TBD) scenario with a maneuvering target is used for simulations. The simulation results indicate that the MM based filter proposed has a good performance in joint detecting and tracking of a weak and maneuvering target, and it is more efficient than the general MM method.

Constrained auxiliary particle filtering for bearings-only maneuvering target tracking

To track the nonlinear,non-Gaussian bearings-only maneuvering target accurately online,the constrained auxiliary particle filtering(CAPF)algorithm is presented.To restrict the samples into the feasible area,the soft measurement constraints are implemented into the update routine via the1 regularization.Meanwhile,to enhance the sampling diversity and efficiency,the target kinetic features and the latest observations are involved into the evolution.To take advantage of the past and the current measurement information simultaneously,the sub-optimal importance distribution is constructed as a Gaussian mixture consisting of the original and modified priors with the fuzzy weighted factors.As a result,the corresponding weights are more evenly distributed,and the posterior distribution of interest is approximated well with a heavier tailor.Simulation results demonstrate the validity and superiority of the CAPF algorithm in terms of efficiency and robustness.

Extended differential geometric guidance law for intercepting maneuvering targets

Without assumptions made on motion states of missile and target, an extended differential geometric guidance law is derived. Through introducing a line of sight rotation coordinate system, the derivation is simplified and has more explicit physical significances. The extended law is theoretically applicable to any engagement scenarios. Then, on basis of the extended law, a modified one is designed without the requirement of target acceleration and an approach is proposed to determining the applied direction of commanded missile acceleration. Qualitative analysis is carried out to study the capture performance and a criterion for capture is given. Simulation results indicate the two laws are effective and make up the deficiency that pure proportional navigation suitable for endoatmospheric interceptions cannot deal with high-speed maneuvering targets. Furthermore, the correctness of the criterion is validated.

A DNN based trajectory optimization method for intercepting non-cooperative maneuvering spacecraft

Current successes in artificial intelligence domain have revitalized interest in neural networks and demonstrated their potential in solving spacecraft trajectory optimization problems. This paper presents a data-free deep neural network(DNN) based trajectory optimization method for intercepting noncooperative maneuvering spacecraft, in a continuous low-thrust scenario. Firstly, the problem is formulated as a standard constrained optimization problem through differential game theory and minimax principle. Secondly, a new DNN is designed to integrate interception dynamic model into the network and involve it in the process of gradient descent, which makes the network endowed with the knowledge of physical constraints and reduces the learning burden of the network. Thus, a DNN based method is proposed, which completely eliminates the demand of training datasets and improves the generalization capacity. Finally, numerical results demonstrate the feasibility and efficiency of our proposed method.

Nonlinear differential geometric guidance for maneuvering target

Based on the idea of zeroing the line of sight rate（LOSR）,a novel nonlinear differential geometric（DG） law for intercepting the agile target is proposed.In the first part,the DG formulations are utilized to describe the relatively kinematics model of missile and target,and the nonlinear DG guidance（DGG） law is proposed based on the nonlinear control theory to eliminate the influence brought by target.Further,the missile guidance commands are derived to overcome the information loss caused by decoupling condition,the new necessary initial condition is developed to guarantee capture the agile target.Then,the designed nonlinear DGG commands are transformed from an arc-length system to the time domain.A desirable aspect of the designed guidance law is that it does not require rigorous information about target acceleration.Representative numerical results show that the designed guidance law obtain a better performance than the traditional DGG law for agile target.