AN EFFECT ITERATION ALGORITHM FOR NUMERICAL SOLUTION OF DISCRETE HAMILTON-JACOBI-BELLMAN EQUATIONS

An algorithm for numerical solution of discrete Hamilton-Jacobi-Bellman equations is proposed. The method begins with a suitable initial guess value of the solution,then finds a suitable matrix to linearize the system and constructs an iteration algorithm to generate the monotone sequence. The convergence of the algorithm for nonlinear discrete Hamilton-Jacobi-Bellman equations is proved. Some numerical examples are presented to confirm the effciency of this algorithm.

Modified domain decomposition method for Hamilton-Jacobi-Bellman equations

This paper presents a modified domain decomposition method for the numerical solution of discrete Hamilton-Jacobi-Bellman equations arising from a class of optimal control problems using diffusion models. A convergence theorem is established. Numerical results indicate the effectiveness and accuracy of the method.

An Iterative Method for Optimal Feedback Control and Generalized HJB Equation

Abstract--In this paper, a new iterative method is proposed to solve the generalized Hamilton-Jacobi-Bellman （GHJB） equation through successively approximate it. Firstly, the GHJB equation is converted to an algebraic equation with the vector norm, which is essentially a set of simultaneous nonlinear equations in the case of dynamic systems. Then, the proposed algorithm solves GHJB equation numerically for points near the origin by considering the linearization of the non-linear equations under a good initial control guess. Finally, the procedure is proved to converge to the optimal stabilizing solution with respect to the iteration variable. In addition, it is shown that the result is a closed-loop control based on this iterative approach. Illustrative examples show that the update control laws will converge to optimal control for nonlinear systems. Index Terms--Generalized Hamilton-Jacobi-Bellman （HJB） equation, iterative method, nonlinear dynamic system, optimal control.

On the Connection between the Hamilton-Jacobi-Bellman and the Fokker-Planck Control Frameworks

In the framework of stochastic processes, the connection between the dynamic programming scheme given by the Hamilton-Jacobi-Bellman equation and a recently proposed control approach based on the Fokker-Planck equation is discussed. Under appropriate assumptions it is shown that the two strategies are equivalent in the case of expected cost functionals, while the Fokker-Planck formalism allows considering a larger classof objectives. To illustratethe connection between the two control strategies, the cases of an Itō stochastic process and of a piecewise-deterministic process are considered.

Adaptive Multi-Step Evaluation Design With Stability Guarantee for Discrete-Time Optimal Learning Control

This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman(HJB)equation.Then,the stability of the system is analyzed using control policies generated by MsHDP.Also,a general stability criterion is designed to determine the admissibility of the current control policy.That is,the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP.Further,based on the convergence and the stability criterion,the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly.Besides,actor-critic is utilized to implement the integrated MsHDP scheme,where neural networks are used to evaluate and improve the iterative policy as the parameter architecture.Finally,two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.

THE OPTIMAL DEDUCTIBLE AND COVERAGE IN INSURANCE CONTRACTS AND EQUILIBRIUM RISK SHARING POLICIES

In this paper, we consider the optimal risk sharing problem between two parties in the insurance business: the insurer and the insured. The risk is allocated between the insurer and the insured by setting a deductible and coverage in the insurance contract. We obtain the optimal deductible and coverage by considering the expected product of the two parties' utilities of terminal wealth according to stochastic optimal control theory. An equilibrium policy is also derived for when there are both a deductible and coverage;this is done by modelling the problem as a stochastic game in a continuous-time framework. A numerical example is provided to illustrate the results of the paper.

Optimal Synchronization Control of Heterogeneous Asymmetric Input-Constrained Unknown Nonlinear MASs via Reinforcement Learning

The asymmetric input-constrained optimal synchronization problem of heterogeneous unknown nonlinear multiagent systems(MASs)is considered in the paper.Intuitively,a state-space transformation is performed such that satisfaction of symmetric input constraints for the transformed system guarantees satisfaction of asymmetric input constraints for the original system.Then,considering that the leader’s information is not available to every follower,a novel distributed observer is designed to estimate the leader’s state using only exchange of information among neighboring followers.After that,a network of augmented systems is constructed by combining observers and followers dynamics.A nonquadratic cost function is then leveraged for each augmented system(agent)for which its optimization satisfies input constraints and its corresponding constrained Hamilton-Jacobi-Bellman(HJB)equation is solved in a data-based fashion.More specifically,a data-based off-policy reinforcement learning(RL)algorithm is presented to learn the solution to the constrained HJB equation without requiring the complete knowledge of the agents’dynamics.Convergence of the improved RL algorithm to the solution to the constrained HJB equation is also demonstrated.Finally,the correctness and validity of the theoretical results are demonstrated by a simulation example.

Controlled Mean-Field Backward Stochastic Differential Equations with Jumps Involving the Value Function

This paper discusses mean-field backward stochastic differentiM equations （mean-field BS- DEs） with jumps and a new type of controlled mean-field BSDEs with jumps, namely mean-field BSDEs with jumps strongly coupled with the value function of the associated control problem. The authors first prove the existence and the uniqueness as well as a comparison theorem for the above two types of BSDEs. For this the authors use an approximation method. Then, with the help of the notion of stochastic backward semigroups introduced by Peng in 1997, the authors get the dynamic programming principle （DPP） for the value functions. Furthermore, the authors prove that the value function is a viscosity solution of the associated nonlocal Hamilton-Jacobi-Bellman （HJB） integro-partial differential equation, which is unique in an adequate space of continuous functions introduced by Barles, et al. in 1997.

Suboptimal Robust Stabilization of Discrete-time Mismatched Nonlinear System

This paper proposes a discrete-time robust control technique for an uncertain nonlinear system. The uncertainty mainly affects the system dynamics due to mismatched parameter variation which is bounded by a predefined known function. In order to compensate the effect of uncertainty, a robust control input is derived by formulating an equivalent optimal control problem for a virtual nominal system with a modified costfunctional. To derive the stabilizing control law for a mismatched system, this paper introduces another control input named as virtual input. This virtual input is not applied directly to stabilize the uncertain system, rather it is used to define a sufficient condition. To solve the nonlinear optimal control problem, a discretetime general Hamilton-Jacobi-Bellman(DT-GHJB) equation is considered and it is approximated numerically through a neural network(NN) implementation. The approximated solution of DTGHJB is used to compute the suboptimal control input for the virtual system. The suboptimal inputs for the virtual system ensure the asymptotic stability of the closed-loop uncertain system. A numerical example is illustrated with simulation results to prove the efficacy of the proposed control algorithm.

DECENTRALIZED CONTROL FOR NONLINEAR DYNAMICAL SYSTEMS: AN L_2-GAIN CONTROL APPROACH

A control method is presented for the problem of decentralized stabilizationof large scale nonlinear systems by designing robust controllers, in the sense of L2-gaincontrol, for each subsystem. An uncertainty tolerance matrix is defined to characterize thedesired robustness leve1 of the overall system. It is then identified that, for a given uncer-tainty tolerance matrix, the design problem is related to the existence of a smooth Positivedefinite solution to a modified Ham ilton -Jacobi - Bellman (H-J-B ) equa tion. The solution,if exists, is exactly the payoff function in terms of the game theory. A decentralized statefeedback law is duly designed, which, under the weak assumption of the zero-state ob-servability on the system, renders the overall closed-loop system aspoptotically stable withan explicitly expressed stability region. Finally, relation between the payoff function andthe uncertainty tolerance matrix is provided, highlighting the 'knowing less and payingmore' philosophy.

A New Convergent Explicit Tree-Grid Method for HJB Equations in One Space Dimension

In this work we introduce a new unconditionally convergent explicit Tree-Grid Method for solving stochastic control problems with one space and one time dimension or equivalently,the corresponding Hamilton-Jacobi-Bellman equation.We prove the convergence of the method and outline the relationships to other numerical methods.The case of vanishing diffusion is treated by introducing an artificial diffusion term.We illustrate the superiority of our method to the standardly used implicit finite difference method on two numerical examples from finance.

A NOTE ON STOCHASTIC OPTIMAL CONTROL OF REFLECTED DIFFUSIONS WITH JUMPS

Stochastic optimal control problems for a class of reflected diffusion with Poisson jumps in a half-space are considered. The nonlinear Nisio' s semigroup for such optimal control problems was constructed. A Hamilton-Jacobi-Bellman equation with the Neumann boundary condition associated with this semigroup was obtained. Then, viscosity solutions of this equation were defined and discussed, and various uniqueness of this equation was also considered. Finally, the value function was such optimal control problems is shown to be a viscosity solution of this equation.

Optimal New Business for Insurer to Minimize the Ruin Probability under Interest Force

Under constant interest force, the risk processes for old and new insurance business are modelled by Brownian motion with drift. By the stochastic control method, the explicit expressions for the minimum ruin probability and the corresponding optimal strategy are derived. Numerical example shows that the minimum probability of ruin and the optimal proportion for new business decrease as the interest rate increases, and vice versa.

Optimal dynamic excess-of-loss reinsurance and multidimensional portfolio selection

In this paper, the surplus process of the insurance company is described by a Brownian motion with drift. In addition, the insurer is allowed to invest in a risk-free asset and n risky assets and purchase excess-of-loss reinsurance. Under short-selling prohibition, we consider two optimization problems: the problem of maximizing the expected exponential utility of terminal wealth and the problem of minimizing the probability of ruin. We first show that the excess-of-loss reinsurance strategy is always better than the proportional reinsurance under two objective functions. Then, by solving the corresponding Hamilton-Jacobi-Bellman equations, the closed-form solutions of their optimal value functions and the corresponding optimal strategies are obtained. In particular, when there is no risky-free interest rate, the results indicate that the optimal strategies, under maximizing the expected exponential utility and minimizing the probability of ruin, are equivalent for some special parameter. This validates Ferguson's longstanding conjecture about the relation between the two problems.

Optimal Proportional Reinsurance for Controlled Risk Process which is Perturbed by Diffusion

In this paper, we study optimal proportional reinsurance policy of an insurer with a risk process which is perturbed by a diffusion. We derive closed-form expressions for the policy and the value function, which are optimal in the sense of maximizing the expected utility in the jump-diffusion framework. We also obtain explicit expressions for the policy and the value function, which are optimal in the sense of maximizing the expected utility or maximizing the survival probability in the diffusion approximation case. Some numerical examples are presented, which show the impact of model parameters on the policy. We also compare the results under the different criteria and different cases.

ASSET-LIABILITY MANAGEMENT UNDER BENCHMARK AND MEAN-VARIANCE CRITERIA IN A JUMP DIFFUSION MARKET

This paper investigates continuous-time asset-liability management under benchmark and mean-variance criteria in a jump diffusion market. Specifically, the authors consider one risk-free asset, one risky asset and one liability, where the risky asset＇s price is governed by an exponential Levy process, the liability evolves according to a Levy process, and there exists a correlation between the risky asset and the liability. Two models are established. One is the benchmark model and the other is the mean-variance model. The benchmark model is solved by employing the stochastic dynamic programming and its results are extended to the mean-variance model by adopting the duality theory. Closed-form solutions of the two models are derived.

Optimal Investment and Premium Control in a Nonlinear Diffusion Model

This paper considers the optimal investment and premium control problem in a diffusion approxi- mation to a non-homogeneous compound Poisson process. In the nonlinear diffusion model, it is assumed that there is an unspecified monotone function describing the relationship between the safety loading of premium and the time-varying claim arrival rate. Hence, in addition to the investment control, the premium rate can be served as a control variable in the optimization problem. Specifically, the problem is investigated in two cases： （i） maximizing the expected utility of terminal wealth, and （ii） minimizing the probability of ruin respectively. In both cases, some properties of the value functions are derived, and closed-form expressions for the optimal policies and the value functions are obtained. The results show that the optimal investment policy and the optimal premium control policy are dependent on each other. Most interestingly, as an example, we show that the nonlinear diffusion model reduces to a diffusion model with a quadratic drift coefficient when the function associated with the premium rate and the claim arrival rate takes a special form. This example shows that the model of study represents a class of nonlinear stochastic control risk model.

Optimal Investment Problem for an Insurer and a Reinsurer

This paper studies the optimal investment problem for an insurer and a reinsurer. The basic claim process is assumed to follow a Brownian motion with drift and the insurer can purchase proportional reinsurance from the reinsurer. The insurer and the reinsurer are allowed to invest in a risk-free asset and a risky asset. Moreover, the authors consider the correlation between the claim process and the price process of the risky asset. The authors first study the optimization problem of maximizing the expected exponential utility of terminal wealth for the insurer. Then with the optimal reinsurance strategy chosen by the insurer, the authors consider two optimization problems for the reinsurer： The problem of maximizing the expected exponential utility of terminal wealth and the problem of minimizing the ruin probability. By solving the corresponding Hamilton-Jacobi-Bellman equations, the authors derive the optimal reinsurance and investment strategies, explicitly. Finally, the authors illustrate the equality of the reinsurer＇s optimal investment strategies under the two cases.

Numerical Solution to Optimal Feedback Control by Dynamic Programming Approach:A Local Approximation Algorithm

This paper considers optimal feedback control for a general continuous time finite-dimensional deterministic system with finite horizon cost functional. A practically feasible algorithm to calculate the numerical solution of the optimal feedback control by dynamic programming approach is developed. The highlights of this algorithm are: a) It is based on a convergent constructive algorithm for optimal feedback control law which was proposed by the authors before through an approximation for the viscosity solution of the time-space discretization scheme developed by dynamic programming method; b) The computation complexity is significantly reduced since only values of viscosity solution on some local cones around the optimal trajectory are calculated. Two numerical experiments are presented to illustrate the effectiveness and fastness of the algorithm.

Numerical simulations for G-Brownian motion

This paper is concerned with numerical simulations for the G- Brownian motion （defined by S. Peng in Stochastic Analysis and Applications, 2007, 541-567）. By the definition of the G-normal distribution, we first show that the G-Brownian motions can be simulated by solving a certain kind of Hamilton-Jacobi-Bellman （HJB） equations. Then, some finite difference methods are designed for the corresponding HJB equations. Numerical simulation results of the G-normal distribution, the G-Brownian motion, and the corresponding quadratic variation process are provided, which characterize basic properties of the G-Brownian motion. We believe that the algorithms in this work serve as a fundamental tool for future studies, e.g., for solving stochastic differential equations （SDEs）/stochastic partial differential equations （SPDEs） driven by the G-Brownian motions.