UAV maneuvering decision-making algorithm based on deep reinforcement learning under the guidance of expert experience

Autonomous umanned aerial vehicle(UAV) manipulation is necessary for the defense department to execute tactical missions given by commanders in the future unmanned battlefield. A large amount of research has been devoted to improving the autonomous decision-making ability of UAV in an interactive environment, where finding the optimal maneuvering decisionmaking policy became one of the key issues for enabling the intelligence of UAV. In this paper, we propose a maneuvering decision-making algorithm for autonomous air-delivery based on deep reinforcement learning under the guidance of expert experience. Specifically, we refine the guidance towards area and guidance towards specific point tasks for the air-delivery process based on the traditional air-to-surface fire control methods.Moreover, we construct the UAV maneuvering decision-making model based on Markov decision processes(MDPs). Specifically, we present a reward shaping method for the guidance towards area and guidance towards specific point tasks using potential-based function and expert-guided advice. The proposed algorithm could accelerate the convergence of the maneuvering decision-making policy and increase the stability of the policy in terms of the output during the later stage of training process. The effectiveness of the proposed maneuvering decision-making policy is illustrated by the curves of training parameters and extensive experimental results for testing the trained policy.

Multiple model PHD filter for tracking sharply maneuvering targets using recursive RANSAC based adaptive birth estimation

An algorithm to track multiple sharply maneuvering targets without prior knowledge about new target birth is proposed. These targets are capable of achieving sharp maneuvers within a short period of time, such as drones and agile missiles.The probability hypothesis density (PHD) filter, which propagates only the first-order statistical moment of the full target posterior, has been shown to be a computationally efficient solution to multitarget tracking problems. However, the standard PHD filter operates on the single dynamic model and requires prior information about target birth distribution, which leads to many limitations in terms of practical applications. In this paper,we introduce a nonzero mean, white noise turn rate dynamic model and generalize jump Markov systems to multitarget case to accommodate sharply maneuvering dynamics. Moreover, to adaptively estimate newborn targets’information, a measurement-driven method based on the recursive random sampling consensus (RANSAC) algorithm is proposed. Simulation results demonstrate that the proposed method achieves significant improvement in tracking multiple sharply maneuvering targets with adaptive birth estimation.

Situational continuity-based air combat autonomous maneuvering decision-making

In order to improve the performance of UAV's autonomous maneuvering decision-making,this paper proposes a decision-making method based on situational continuity.The algorithm in this paper designs a situation evaluation function with strong guidance,then trains the Long Short-Term Memory(LSTM)under the framework of Deep Q Network(DQN)for air combat maneuvering decision-making.Considering the continuity between adjacent situations,the method takes multiple consecutive situations as one input of the neural network.To reflect the difference between adjacent situations,the method takes the difference of situation evaluation value as the reward of reinforcement learning.In different scenarios,the algorithm proposed in this paper is compared with the algorithm based on the Fully Neural Network(FNN)and the algorithm based on statistical principles respectively.The results show that,compared with the FNN algorithm,the algorithm proposed in this paper is more accurate and forwardlooking.Compared with the algorithm based on the statistical principles,the decision-making of the algorithm proposed in this paper is more efficient and its real-time performance is better.

Optimal maneuvering strategy of spacecraft evasion based on angles-only measurement and observability analysis

Spacecraft orbit evasion is an effective method to ensure space safety. In the spacecraft’s orbital plane, the space non-cooperate target with autonomous approaching to the spacecraft may have a dangerous rendezvous. To deal with this problem, an optimal maneuvering strategy based on the relative navigation observability degree is proposed with angles-only measurements. A maneuver evasion relative navigation model in the spacecraft’s orbital plane is constructed and the observability measurement criteria with process noise and measurement noise are defined based on the posterior Cramer-Rao lower bound. Further, the optimal maneuver evasion strategy in spacecraft’s orbital plane based on the observability is proposed. The strategy provides a new idea for spacecraft to evade safety threats autonomously. Compared with the spacecraft evasion problem based on the absolute navigation, more accurate evasion results can be obtained. The simulation indicates that this optimal strategy can weaken the system’s observability and reduce the state estimation accuracy of the non-cooperative target, making it impossible for the non-cooperative target to accurately approach the spacecraft.

CFD Investigations of Ship Maneuvering in Waves Using naoe-FOAM-SJTU Solver

Ship maneuvering in waves includes the performance of ship resistance, seakeeping, propulsion, and maneuverability. It is a complex hydrodynamic problem with the interaction of many factors. With the purpose of directly predicting the behavior of ship maneuvering in waves, a CFD solver named naoe-FOAM-SJTU is developed by the Computational Marine Hydrodynamics Lab(CMHL) in Shanghai Jiao Tong University. The solver is based on open source platform OpenFOAM and has introduced dynamic overset grid technology to handle complex ship hull-propeller-rudder motion system. Maneuvering control module based on feedback control mechanism is also developed to accurately simulate corresponding motion behavior of free running ship maneuver. Inlet boundary wavemaker and relaxation zone technique is used to generate desired waves. Based on the developed modules, unsteady Reynolds-averaged Navier-Stokes(RANS) computations are carried out for several validation cases of free running ship maneuver in waves including zigzag, turning circle, and course keeping maneuvers. The simulation results are compared with available benchmark data. Ship motions, trajectories, and other maneuvering parameters are consistent with available experimental data, which indicate that the present solver can be suitable and reliable in predicting the performance of ship maneuvering in waves. Flow visualizations, such as free surface elevation, wake flow, vortical structures, are presented to explain the hydrodynamic performance of ship maneuvering in waves. Large flow separation can be observed around propellers and rudders. It is concluded that RANS approach is not accurate enough for predicting ship maneuvering in waves with large flow separations and detached eddy simulation(DES) or large eddy simulation(LES) computations are required to improve the prediction accuracy.

Maneuvering target tracking of UAV based on MN-DDPG and transfer learning

Tracking maneuvering target in real time autonomously and accurately in an uncertain environment is one of the challenging missions for unmanned aerial vehicles(UAVs).In this paper,aiming to address the control problem of maneuvering target tracking and obstacle avoidance,an online path planning approach for UAV is developed based on deep reinforcement learning.Through end-to-end learning powered by neural networks,the proposed approach can achieve the perception of the environment and continuous motion output control.This proposed approach includes:(1)A deep deterministic policy gradient(DDPG)-based control framework to provide learning and autonomous decision-making capability for UAVs;(2)An improved method named MN-DDPG for introducing a type of mixed noises to assist UAV with exploring stochastic strategies for online optimal planning;and(3)An algorithm of taskdecomposition and pre-training for efficient transfer learning to improve the generalization capability of UAV’s control model built based on MN-DDPG.The experimental simulation results have verified that the proposed approach can achieve good self-adaptive adjustment of UAV’s flight attitude in the tasks of maneuvering target tracking with a significant improvement in generalization capability and training efficiency of UAV tracking controller in uncertain environments.

An impact angle constraint integral sliding mode guidance law for maneuvering targets interception

An integral sliding mode guidance law(ISMGL)combined with the advantages of the integral sliding mode control(SMC)method is designed to address maneuvering target interception problems with impact angle constraints.The relative motion equation of the missile and the target considering the impact angle constraint is established in the longitudinal plane,and an integral sliding mode surface is constructed.The proposed guidance law resolves the existence of a steady-state error problem in the traditional SMC.Such a guidance law ensures that the missile hits the target with an ideal impact angle in finite time and the missile is kept highly robust throughout the interception process.By adopting the dynamic surface control method,the ISMGL is designed considering the impact angle constraints and the autopilot dynamic characteristics.According to the Lyapunov stability theorem,all states of the closed-loop system are finally proven to be uniformly bounded.Simulation results are compared with the general sliding mode guidance law and the trajectory shaping guidance law,and the findings verify the effectiveness and superiority of the ISMGL.

Modified unscented Kalman filter using modified filter gain and variance scale factor for highly maneuvering target tracking

To improve the low tracking precision caused by lagged filter gain or imprecise state noise when the target highly maneuvers, a modified unscented Kalman filter algorithm based on the improved filter gain and adaptive scale factor of state noise is presented. In every filter process, the estimated scale factor is used to update the state noise covariance Qk, and the improved filter gain is obtained in the filter process of unscented Kalman filter （UKF） via predicted variance Pk｜k-1, which is similar to the standard Kalman filter. Simulation results show that the proposed algorithm provides better accuracy and ability to adapt to the highly maneuvering target compared with the standard UKF.

Adaptive Maneuvering Frequency Method of Current Statistical Model

Current statistical model(CSM) has a good performance in maneuvering target tracking. However, the fixed maneuvering frequency will deteriorate the tracking results, such as a serious dynamic delay, a slowly converging speedy and a limited precision when using Kalman filter(KF) algorithm. In this study, a new current statistical model and a new Kalman filter are proposed to improve the performance of maneuvering target tracking. The new model which employs innovation dominated subjection function to adaptively adjust maneuvering frequency has a better performance in step maneuvering target tracking, while a fluctuant phenomenon appears. As far as this problem is concerned, a new adaptive fading Kalman filter is proposed as well. In the new Kalman filter, the prediction values are amended in time by setting judgment and amendment rules,so that tracking precision and fluctuant phenomenon of the new current statistical model are improved. The results of simulation indicate the effectiveness of the new algorithm and the practical guiding significance.

Sensor scheduling for ground maneuvering target tracking in presence of detection blind zone

Continuous and stable tracking of the ground maneuvering target is a challenging problem due to the complex terrain and high clutter. A collaborative tracking method of the multisensor network is presented for the ground maneuvering target in the presence of the detection blind zone(DBZ). First, the sensor scheduling process is modeled within the partially observable Markov decision process(POMDP) framework. To evaluate the target tracking accuracy of the sensor, the Fisher information is applied to constructing the reward function. The key of the proposed scheduling method is forecasting and early decisionmaking. Thus, an approximate method based on unscented sampling is presented to estimate the target state and the multi-step scheduling reward over the prediction time horizon. Moreover, the problem is converted into a nonlinear optimization problem, and a fast search algorithm is given to solve the sensor scheduling scheme quickly. Simulation results demonstrate the proposed nonmyopic scheduling method(Non-MSM) has a better target tracking accuracy compared with traditional methods.

Sequential maneuvering decisions based on multi-stage influence diagram in air combat

A multi-stage influence diagram is used to model the pilot＇s sequential decision making in one on one air combat. The model based on the multi-stage influence diagram graphically describes the elements of decision process, and contains a point-mass model for the dynamics of an aircraft and takes into account the decision maker＇s preferences under uncertain conditions. Considering an active opponent, the opponent＇s maneuvers can be modeled stochastically. The solution of multistage influence diagram can be obtained by converting the multistage influence diagram into a two-level optimization problem. The simulation results show the model is effective.

Multiple model efficient particle filter based track-before-detect for maneuvering weak targets

It is a tough problem to jointly detect and track a weak target, and it becomes even more challenging when the target is maneuvering. The above problem is formulated by using the Bayesian theory and a multiple model(MM) based filter is proposed. The filter presented uses the MM method to accommodate the multiple motions that a maneuvering target may travel under by adding a random variable representing the motion model to the target state. To strengthen the efficiency performance of the filter,the target existence variable is separated from the target state and the existence probability is calculated in a more efficient way. To examine the performance of the MM based approach, a typical track-before-detect(TBD) scenario with a maneuvering target is used for simulations. The simulation results indicate that the MM based filter proposed has a good performance in joint detecting and tracking of a weak and maneuvering target, and it is more efficient than the general MM method.

Constrained auxiliary particle filtering for bearings-only maneuvering target tracking

To track the nonlinear,non-Gaussian bearings-only maneuvering target accurately online,the constrained auxiliary particle filtering(CAPF)algorithm is presented.To restrict the samples into the feasible area,the soft measurement constraints are implemented into the update routine via the1 regularization.Meanwhile,to enhance the sampling diversity and efficiency,the target kinetic features and the latest observations are involved into the evolution.To take advantage of the past and the current measurement information simultaneously,the sub-optimal importance distribution is constructed as a Gaussian mixture consisting of the original and modified priors with the fuzzy weighted factors.As a result,the corresponding weights are more evenly distributed,and the posterior distribution of interest is approximated well with a heavier tailor.Simulation results demonstrate the validity and superiority of the CAPF algorithm in terms of efficiency and robustness.

A DNN based trajectory optimization method for intercepting non-cooperative maneuvering spacecraft

Current successes in artificial intelligence domain have revitalized interest in neural networks and demonstrated their potential in solving spacecraft trajectory optimization problems. This paper presents a data-free deep neural network(DNN) based trajectory optimization method for intercepting noncooperative maneuvering spacecraft, in a continuous low-thrust scenario. Firstly, the problem is formulated as a standard constrained optimization problem through differential game theory and minimax principle. Secondly, a new DNN is designed to integrate interception dynamic model into the network and involve it in the process of gradient descent, which makes the network endowed with the knowledge of physical constraints and reduces the learning burden of the network. Thus, a DNN based method is proposed, which completely eliminates the demand of training datasets and improves the generalization capacity. Finally, numerical results demonstrate the feasibility and efficiency of our proposed method.

Nonlinear differential geometric guidance for maneuvering target

Based on the idea of zeroing the line of sight rate（LOSR）,a novel nonlinear differential geometric（DG） law for intercepting the agile target is proposed.In the first part,the DG formulations are utilized to describe the relatively kinematics model of missile and target,and the nonlinear DG guidance（DGG） law is proposed based on the nonlinear control theory to eliminate the influence brought by target.Further,the missile guidance commands are derived to overcome the information loss caused by decoupling condition,the new necessary initial condition is developed to guarantee capture the agile target.Then,the designed nonlinear DGG commands are transformed from an arc-length system to the time domain.A desirable aspect of the designed guidance law is that it does not require rigorous information about target acceleration.Representative numerical results show that the designed guidance law obtain a better performance than the traditional DGG law for agile target.

ISAR autofocus imaging algorithm for maneuvering targets based on deep learning and keystone transform

The issue of small-angle maneuvering targets inverse synthetic aperture radar(ISAR)imaging has been successfully addressed by popular motion compensation algorithms.However,when the target’s rotational velocity is sufficiently high during the dwell time of the radar,such compensation algorithms cannot obtain a high quality image.This paper proposes an ISAR imaging algorithm based on keystone transform and deep learning algorithm.The keystone transform is used to coarsely compensate for the target’s rotational motion and translational motion,and the deep learning algorithm is used to achieve a super-resolution image.The uniformly distributed point target data are used as the data set of the training u-net network.In addition,this method does not require estimating the motion parameters of the target,which simplifies the algorithm steps.Finally,several experiments are performed to demonstrate the effectiveness of the proposed algorithm.

Extended differential geometric guidance law for intercepting maneuvering targets

Without assumptions made on motion states of missile and target, an extended differential geometric guidance law is derived. Through introducing a line of sight rotation coordinate system, the derivation is simplified and has more explicit physical significances. The extended law is theoretically applicable to any engagement scenarios. Then, on basis of the extended law, a modified one is designed without the requirement of target acceleration and an approach is proposed to determining the applied direction of commanded missile acceleration. Qualitative analysis is carried out to study the capture performance and a criterion for capture is given. Simulation results indicate the two laws are effective and make up the deficiency that pure proportional navigation suitable for endoatmospheric interceptions cannot deal with high-speed maneuvering targets. Furthermore, the correctness of the criterion is validated.

Maneuvering Modeling and Simulation of AUV Dynamic Systems with Euler-Rodriguez Quaternion Method

The purpose of this study is to develop maneuvering models and systems of a simulator to improve the motion performance of autonomous underwater vehicles （AUVs） at the preliminary design stages in advance. The AUVs simulation systems based on the standard submarine equations of motion in six-degree-of-freedom （6-DOF） integrated with the Euler-Rodriguez quaternion method for representing singularity-free AUV attitude and time-saving calculation, and with a nonlinear control model for maneuvering and depth control simulations, time-marching in the fourth-order Runge-Kutta scheme. For validation of the simulation codes, results of the ISiMI AUV open-loop tests including turning test and zigzag test as well as an AUV simulator on the basis of Euler-angle method were used to compare with the quaternion-based AUV simulator. The computational results from the proposed simulator agree well with those from both the ISiMI AUV experiments and the Euler-angle based simulations. Additionally, a new maneuvering procedure, namely ＂put-out＂ was implemented to test directional stability for a large-scale AUV in the proposed AUV simulator that can be considered for vehicles in space as well as in constrained planes.

Multi-Innovation Gradient Iterative Locally Weighted Learning Identification for A Nonlinear Ship Maneuvering System

This paper explores a highly accurate identification modeling approach for the ship maneuvering motion with fullscale trial. A multi-innovation gradient iterative(MIGI) approach is proposed to optimize the distance metric of locally weighted learning(LWL), and a novel non-parametric modeling technique is developed for a nonlinear ship maneuvering system. This proposed method’s advantages are as follows: first, it can avoid the unmodeled dynamics and multicollinearity inherent to the conventional parametric model; second, it eliminates the over-learning or underlearning and obtains the optimal distance metric; and third, the MIGI is not sensitive to the initial parameter value and requires less time during the training phase. These advantages result in a highly accurate mathematical modeling technique that can be conveniently implemented in applications. To verify the characteristics of this mathematical model, two examples are used as the model platforms to study the ship maneuvering.

Highly maneuvering target tracking using multi-parameter fusion Singer model

An algorithm of highly maneuvering target tracking is proposed to solve the problem of large tracking error caused by strong maneuver. In this algorithm, a new estimator, named as multi-parameter fusion Singer (MF-Singer) model is derived based on the Singer model and the fuzzy reasoning method by using radial acceleration and velocity of the target, and applied to the problem of maneuvering target tracking in strong maneuvering environment and operating environment. The tracking performance of the MF-Singer model is evaluated and compared with other manuevering tracking models. It is shown that the MF-Singer model outperforms these algorithms in several examples.