Using approximate dynamic programming for multi-ESM scheduling to track ground moving targets

This paper researches the adaptive scheduling problem of multiple electronic support measures(multi-ESM) in a ground moving radar targets tracking application. It is a sequential decision-making problem in uncertain environment. For adaptive selection of appropriate ESMs, we generalize an approximate dynamic programming(ADP) framework to the dynamic case. We define the environment model and agent model, respectively. To handle the partially observable challenge, we apply the unsented Kalman filter(UKF) algorithm for belief state estimation. To reduce the computational burden, a simulation-based approach rollout with a redesigned base policy is proposed to approximate the long-term cumulative reward. Meanwhile, Monte Carlo sampling is combined into the rollout to estimate the expectation of the rewards. The experiments indicate that our method outperforms other strategies due to its better performance in larger-scale problems.

Approximate Dynamic Programming for Stochastic Resource Allocation Problems

A stochastic resource allocation model, based on the principles of Markov decision processes(MDPs), is proposed in this paper. In particular, a general-purpose framework is developed, which takes into account resource requests for both instant and future needs. The considered framework can handle two types of reservations(i.e., specified and unspecified time interval reservation requests), and implement an overbooking business strategy to further increase business revenues. The resulting dynamic pricing problems can be regarded as sequential decision-making problems under uncertainty, which is solved by means of stochastic dynamic programming(DP) based algorithms. In this regard, Bellman’s backward principle of optimality is exploited in order to provide all the implementation mechanisms for the proposed reservation pricing algorithm. The curse of dimensionality, as the inevitable issue of the DP both for instant resource requests and future resource reservations,occurs. In particular, an approximate dynamic programming(ADP) technique based on linear function approximations is applied to solve such scalability issues. Several examples are provided to show the effectiveness of the proposed approach.

Policy iteration optimal tracking control for chaotic systems by using an adaptive dynamic programming approach

A policy iteration algorithm of adaptive dynamic programming（ADP） is developed to solve the optimal tracking control for a class of discrete-time chaotic systems. By system transformations, the optimal tracking problem is transformed into an optimal regulation one. The policy iteration algorithm for discrete-time chaotic systems is first described. Then,the convergence and admissibility properties of the developed policy iteration algorithm are presented, which show that the transformed chaotic system can be stabilized under an arbitrary iterative control law and the iterative performance index function simultaneously converges to the optimum. By implementing the policy iteration algorithm via neural networks,the developed optimal tracking control scheme for chaotic systems is verified by a simulation.

Two-stage robust optimization of power cost minimization problem in gunbarrel natural gas networks by approximate dynamic programming

In short-term operation of natural gas network,the impact of demand uncertainty is not negligible.To address this issue we propose a two-stage robust model for power cost minimization problem in gunbarrel natural gas networks.The demands between pipelines and compressor stations are uncertain with a budget parameter,since it is unlikely that all the uncertain demands reach the maximal deviation simultaneously.During solving the two-stage robust model we encounter a bilevel problem which is challenging to solve.We formulate it as a multi-dimensional dynamic programming problem and propose approximate dynamic programming methods to accelerate the calculation.Numerical results based on real network in China show that we obtain a speed gain of 7 times faster in average without compromising optimality compared with original dynamic programming algorithm.Numerical results also verify the advantage of robust model compared with deterministic model when facing uncertainties.These findings offer short-term operation methods for gunbarrel natural gas network management to handle with uncertainties.

Chaotic system optimal tracking using data-based synchronous method with unknown dynamics and disturbances

We develop an optimal tracking control method for chaotic system with unknown dynamics and disturbances. The method allows the optimal cost function and the corresponding tracking control to update synchronously. According to the tracking error and the reference dynamics, the augmented system is constructed. Then the optimal tracking control problem is defined. The policy iteration （PI） is introduced to solve the rain-max optimization problem. The off-policy adaptive dynamic programming （ADP） algorithm is then proposed to find the solution of the tracking Hamilton-Jacobi- Isaacs （HJI） equation online only using measured data and without any knowledge about the system dynamics. Critic neural network （CNN）, action neural network （ANN）, and disturbance neural network （DNN） are used to approximate the cost function, control, and disturbance. The weights of these networks compose the augmented weight matrix, and the uniformly ultimately bounded （UUB） of which is proven. The convergence of the tracking error system is also proven. Two examples are given to show the effectiveness of the proposed synchronous solution method for the chaotic system tracking problem.

Call for papers Journal of Control Theory and Applications Special issue on Approximate dynamic programming and reinforcement learning

Approximate dynamic programming (ADP) is a general and effective approach for solving optimal control and estimation problems by adapting to uncertain and nonconvex environments over time.

CONVERGENCE OF THE APPROXIMATE SOLUTIONS TO ISENTROPIC GAS DYNAMICS

This paper gives four pairs of entropies (η_i, q_i) (i=1, 2, 3, 4) to the isentropic gas dynamics equations ρ_t+(ρu)_x=0 (ρu)_t+(ρu^2+p(ρ))_x=0 p(ρ)=k^2ρ~γ,1<γ<3。 when all the function equations are satisfied

Capacity planning of hydro-wind-solar hybrid power systems considering hydropower forbidden zones

In the capacity planning of hydro-wind-solar power systems(CPHPS),it is crucial to use flexible hydropower to complement the variable wind-solar power.Hydropower units must be operated such that they avoid specific restricted operation zones,that is,forbidden zones(FZs),to avoid the risks associated with hydropower unit vibration.FZs cause limitations in terms of both the hydropower generation and flexible regulation in the hydro-wind-solar power systems.Therefore,it is essential to consider FZs when determining the optimal wind-solar power capacity that can be compensated by the hydropower.This study presents a mathematical model that incorporates the FZ constraints into the CPHPS problem.Firstly,the FZs of the hydropower units are converted into those of the hydropower plants based on set theory.Secondly,a mathematical model was formulated for the CPHPS,which couples the FZ constraints of hydropower plants with other operational constraints(e.g.,power balance constraints,new energy consumption limits,and hydropower generation functions).Thirdly,dynamic programming with successive approximations is employed to solve the proposed model.Lastly,case studies were conducted on the hydro-wind-solar system of the Qingshui River to demonstrate the effectiveness of the proposed model.

Discounted Iterative Adaptive Critic Designs With Novel Stability Analysis for Tracking Control

The core task of tracking control is to make the controlled plant track a desired trajectory.The traditional performance index used in previous studies cannot eliminate completely the tracking error as the number of time steps increases.In this paper,a new cost function is introduced to develop the value-iteration-based adaptive critic framework to solve the tracking control problem.Unlike the regulator problem,the iterative value function of tracking control problem cannot be regarded as a Lyapunov function.A novel stability analysis method is developed to guarantee that the tracking error converges to zero.The discounted iterative scheme under the new cost function for the special case of linear systems is elaborated.Finally,the tracking performance of the present scheme is demonstrated by numerical results and compared with those of the traditional approaches.

Stochastic resonance in coupled weakly-damped bistable oscillators subjected to additive and multiplicative noises

With coupled weakly-damped periodically driven bistable oscillators subjected to additive and multiplicative noises under concern, the objective of this paper is to check to what extent the resonant point predicted by the Gaussian distribution assumption can approximate the simulated one. The investigation based on the dynamical mean-field approx- imation and the direct simulation demonstrates that the pre- dicted resonant point and the simulated one are basically co- incident for the case of pure additive noise, but for the case including multiplicative noise the situation becomes some- what complex. Specifically speaking, when stochastic res- onance （SR） is observed by changing the additive noise in- tensity, the predicted resonant point is lower than the sim- ulated one; nevertheless, when SR is observed by chang- ing the multiplicative noise intensity, the predicted resonant point is higher than the simulated one. Our observations im- ply that the Gaussian distribution assumption can not exactly describe the actual situation, but it is useful to some extent in predicting the low-frequency stochastic resonance of the coupled weakly-damped bistable oscillator.

A Novel Distributed Optimal Adaptive Control Algorithm for Nonlinear Multi-Agent Differential Graphical Games

In this paper, an online optimal distributed learning algorithm is proposed to solve leader-synchronization problem of nonlinear multi-agent differential graphical games. Each player approximates its optimal control policy using a single-network approximate dynamic programming(ADP) where only one critic neural network(NN) is employed instead of typical actorcritic structure composed of two NNs. The proposed distributed weight tuning laws for critic NNs guarantee stability in the sense of uniform ultimate boundedness(UUB) and convergence of control policies to the Nash equilibrium. In this paper, by introducing novel distributed local operators in weight tuning laws, there is no more requirement for initial stabilizing control policies. Furthermore, the overall closed-loop system stability is guaranteed by Lyapunov stability analysis. Finally, Simulation results show the effectiveness of the proposed algorithm.

A Novel Face Recognition Algorithm for Distinguishing Faces with Various Angles

In order to distinguish faces of various angles during face recognition, an algorithm of the combination of approximate dynamic programming （ADP） called action dependent heuristic dynamic programming （ADHDP） and particle swarm optimization （PSO） is presented. ADP is used for dynamically changing the values of the PSO parameters. During the process of face recognition, the discrete cosine transformation （DCT） is first introduced to reduce negative effects. Then, Karhunen-Loeve （K-L） transformation can be used to compress images and decrease data dimensions. According to principal component analysis （PCA）, the main parts of vectors are extracted for data representation. Finally, radial basis function （RBF） neural network is trained to recognize various faces. The training of RBF neural network is exploited by ADP-PSO. In terms of ORL Face Database, the experimental result gives a clear view of its accurate efficiency.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems,in this paper,a new iterative adaptive dynamic programming algorithm,which is the discrete-time time-varying policy iteration(DTTV)algorithm,is developed.The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance.The admissibility of the iterative control law is analyzed.The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution.To implement the algorithm,neural networks are employed and a new implementation structure is established,which avoids solving the generalized Bellman equation in each iteration.Finally,the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm,where the mass and pendulum bar length are permitted to be time-varying parameters.The effectiveness of the developed method is illustrated by numerical results and comparisons.

Quantum phase transitions of fermionic atoms in an anisotropic triangular optical lattice

The effect of anisotropy caused by a confining potential on the properties of fermionic cold atoms in a triangular optical lattice is systematically investigated by using the dynamical cluster approximation combined with the continuous time quantum Monte-Carlo algorithm. The quantum phase diagrams which reflect the temperature-interaction relation and the competition between the anisotropic parameter and the interaction are presented with full consideration of the anisotropy of the system. Our results show that the system undergoes a transition from Fermi liquid to Mott insulator when the repulsive interaction reaches a critical value. The Kondo effect also can be observed in this system and the pseudogap is suppressed at low temperatures due to the Kondo effect. A feasible experiment protocol to observe these phenomena in an anisotropic triangular optical lattice with cold atoms is proposed, in which the hopping terms are closely related to the lattice confining potential and the atomic interaction can be adjusted via the Feshbach resonance.

Off-policy integral reinforcement learning optimal tracking control for continuous-time chaotic systems

This paper estimates an off-policy integral reinforcement learning（IRL） algorithm to obtain the optimal tracking control of unknown chaotic systems. Off-policy IRL can learn the solution of the HJB equation from the system data generated by an arbitrary control. Moreover, off-policy IRL can be regarded as a direct learning method, which avoids the identification of system dynamics. In this paper, the performance index function is first given based on the system tracking error and control error. For solving the Hamilton–Jacobi–Bellman（HJB） equation, an off-policy IRL algorithm is proposed.It is proven that the iterative control makes the tracking error system asymptotically stable, and the iterative performance index function is convergent. Simulation study demonstrates the effectiveness of the developed tracking control method.

Effect of spatially correlated noise on stochastic synchronization in globally coupled FitzHugh–Nagumo neuron systems

The phenomenon of stochastic synchronization in globally coupled FitzHugh–Nagumo（FHN） neuron system subjected to spatially correlated Gaussian noise is investigated based on dynamical mean-field approximation（DMA） and direct simulation（DS）. Results from DMA are in good quantitative or qualitative agreement with those from DS for weak noise intensity and larger system size. Whether the consisting single FHN neuron is staying at the resting state, subthreshold oscillatory regime, or the spiking state, our investigation shows that the synchronization ratio of the globally coupled system becomes higher as the noise correlation coefficient increases, and thus we conclude that spatial correlation has an active effect on stochastic synchronization, and the neurons can achieve complete synchronization in the sense of statistics when the noise correlation coefficient tends to one. Our investigation also discloses that the noise spatial correlation plays the same beneficial role as the global coupling strength in enhancing stochastic synchronization in the ensemble. The result might be useful in understanding the information coding mechanism in neural systems.

Distributed dynamic stochastic approximation algorithm over time-varying networks

In this paper,a distributed stochastic approximation algorithm is proposed to track the dynamic root of a sum of time-varying regression functions over a network.Each agent updates its estimate by using the local observation,the dynamic information of the global root,and information received from its neighbors.Compared with similar works in optimization area,we allow the observation to be noise-corrupted,and the noise condition is much weaker.Furthermore,instead of the upper bound of the estimate error,we present the asymptotic convergence result of the algorithm.The consensus and convergence of the estimates are established.Finally,the algorithm is applied to a distributed target tracking problem and the numerical example is presented to demonstrate the performance of the algorithm.

Real-time Risk-averse Dispatch of an Integrated Electricity and Natural Gas System via Condi-tional Value-at-risk-based Lookup-table Ap-proximate Dynamic Programming

The real-time risk-averse dispatch problem of an integrated electricity and natural gas system(IEGS)is studied in this paper.It is formulated as a real-time conditional value-at-risk(CVaR)-based risk-averse dis-patch model in the Markov decision process framework.Because of its stochasticity,nonconvexity and nonlinearity,the model is difficult to analyze by traditional algorithms in an acceptable time.To address this non-deterministic polynomial-hard problem,a CVaR-based lookup-table approximate dynamic programming(CVaR-ADP)algo-rithm is proposed,and the risk-averse dispatch problem is decoupled into a series of tractable subproblems.The line pack is used as the state variable to describe the impact of one period’s decision on the future.This facilitates the reduction of load shedding and wind power curtailment.Through the proposed method,real-time decisions can be made according to the current information,while the value functions can be used to overview the whole opti-mization horizon to balance the current cost and future risk loss.Numerical simulations indicate that the pro-posed method can effectively measure and control the risk costs in extreme scenarios.Moreover,the decisions can be made within 10 s,which meets the requirement of the real-time dispatch of an IEGS.Index Terms—Integrated electricity and natural gas system,approximate dynamic programming,real-time dispatch,risk-averse,conditional value-at-risk.

Transfer-based Approximate Dynamic Programmingfor Rolling Security-constrained Unit Commitment with Uncertainties

This paper studies the rolling security-constrained unit commitment(RSCUC)problem with AC power flow and uncertainties.For this NP-hard problem,it is modeled as a Markov decision process,which is then solved by a transfer-based approximate dynamic programming(TADP)algorithm proposed in this paper.Different from traditional approximate dynamic programming(ADP)algorithms,TADP can obtain the commitment states of most units in advance through a decision transfer technique,thus reducing the action space of TADP significantly.Moreover,compared with traditional ADP algorithms,which require to determine the commitment state of each unit,TADP only needs determine the unit with the smallest on-state probability among all on-state units,thus further reducing the action space.The proposed algorithm can also prevent the iter-ative update of value functions and the reliance on rolling forecast information,which makes more sense in the rolling decision-making process of RSCUC.Finally,nu-merical simulations are carried out on a modified IEEE 39-bus system and a real 2778-bus system to demonstrate the effectiveness of the proposed algorithm.

Adaptive dynamic programming for online solution of a zero-sum differential game

This paper will present an approximate/adaptive dynamic programming(ADP) algorithm,that uses the idea of integral reinforcement learning(IRL),to determine online the Nash equilibrium solution for the two-player zerosum differential game with linear dynamics and infinite horizon quadratic cost.The algorithm is built around an iterative method that has been developed in the control engineering community for solving the continuous-time game algebraic Riccati equation(CT-GARE),which underlies the game problem.We here show how the ADP techniques will enhance the capabilities of the offline method allowing an online solution without the requirement of complete knowledge of the system dynamics.The feasibility of the ADP scheme is demonstrated in simulation for a power system control application.The adaptation goal is the best control policy that will face in an optimal manner the highest load disturbance.