Optimal Reservoir Operation Using Stochastic Dynamic Programming

This paper focused on the applying stochastic dynamic programming (SDP) to reservoir operation. Based on the two stages decision procedure, we built an operation model for reservoir operation to derive operating rules. With a case study of the China’s Three Gorges Reservoir, long-term operating rules are obtained. Based on the derived operating rules, the reservoir is simulated with the inflow from 1882 to 2005, which the mean hydropower generation is 85.71 billion kWh. It is shown that the SDP works well in the reservoir operation.

Experimental Study of Methods of Scenario Lattice Construction for Stochastic Dual Dynamic Programming

The stochastic dual dynamic programming (SDDP) algorithm is becoming increasingly used. In this paper we present analysis of different methods of lattice construction for SDDP exemplifying a realistic variant of the newsvendor problem, incorporating storage of production. We model several days of work and compare the profits realized using different methods of the lattice construction and the corresponding computer time spent in lattice construction. Our case differs from the known one because we consider not only a multidimensional but also a multistage case with stage dependence. We construct scenario lattice for different Markov processes which play a crucial role in stochastic modeling. The novelty of our work is comparing different methods of scenario lattice construction. We considered a realistic variant of the newsvendor problem. The results presented in this article show that the Voronoi method slightly outperforms others, but the k-means method is much faster overall.

Relationship between Maximum Principle and Dynamic Programming in Stochastic Differential Games and Applications

This paper is concerned with the relationship between maximum principle and dynamic programming in zero-sum stochastic differential games. Under the assumption that the value function is enough smooth, relations among the adjoint processes, the generalized Hamiltonian function and the value function are given. A portfolio optimization problem under model uncertainty in the financial market is discussed to show the applications of our result.

A NEW DETERMINISTIC FORMULATION FOR DYNAMIC STOCHASTIC PROGRAMMING PROBLEMS AND ITS NUMERICAL COMPARISON WITH OTHERS

A new deterministic formulation,called the conditional expectation formulation,is proposed for dynamic stochastic programming problems in order to overcome some disadvantages of existing deterministic formulations.We then check the impact of the new deterministic formulation and other two deterministic formulations on the corresponding problem size,nonzero elements and solution time by solving some typical dynamic stochastic programming problems with different interior point algorithms.Numerical results show the advantage and application of the new deterministic formulation.

Energy Management of Price-maker Community Energy Storage by Stochastic Dynamic Programming

In this paper,we propose an analytical stochastic dynamic programming(SDP)algorithm to address the optimal management problem of price-maker community energy storage.As a price-maker,energy storage smooths price differences,thus decreasing energy arbitrage value.However,this price-smoothing effect can result in significant external welfare changes by reduc-ing consumer costs and producer revenues,which is not negligible for the community with energy storage systems.As such,we formulate community storage management as an SDP that aims to maximize both energy arbitrage and community welfare.To incorporate market interaction into the SDP format,we propose a framework that derives partial but sufficient market information to approximate impact of storage operations on market prices.Then we present an analytical SDP algorithm that does not require state discretization.Apart from computational efficiency,another advantage of the analytical algorithm is to guide energy storage to charge/discharge by directly comparing its current marginal value with expected future marginal value.Case studies indicate community-owned energy storage that maximizes both arbitrage and welfare value gains more benefits than storage that maximizes only arbitrage.The proposed algorithm ensures optimality and largely reduces the computational complexity of the standard SDP.Index Terms-Analytical stochastic dynamic programming,energy management,energy storage,price-maker,social welfare.

Performance-based global reliability assessment of a high-rise frame-core tube structure subjected to multi-dimensional stochastic earthquakes

When evaluating the seismic safety and reliability of complex engineering structures,it is a critical problem to reasonably consider the randomness and multi-dimensional nature of ground motions.To this end,a proposed modeling strategy of multi-dimensional stochastic earthquakes is addressed in this study.This improved seismic model has several merits that enable it to better provide seismic analyses of structures.Specifically,at first,the ground motion model is compatible with the design response spectrum.Secondly,the evolutionary power spectrum involved in the model and the design response spectrum are constructed accordingly with sufficient consideration of the correlation between different seismic components.Thirdly,the random function-based dimension-reduction representation is applied,by which seismic modeling is established,with three elementary random variables.Numerical simulations of multi-dimensional stochastic ground motions in a specific design scenario indicate the effectiveness of the proposed modeling strategy.Moreover,the multi-dimensional seismic response and the global reliability of a high-rise frame-core tube structure is discussed in detail to further illustrate the engineering applicability of the proposed method.The analytical investigations demonstrate that the suggested stochastic model of multi-dimensional ground motion is available for accurate seismic response analysis and dynamic reliability assessment of complex engineering structures for performance-based seismic resistance design.

Two-stage stochastic approach for spinning reserve allocation in dynamic economic dispatch

A novel approach was proposed to allocate spinning reserve for dynamic economic dispatch.The proposed approach set up a two-stage stochastic programming model to allocate reserve.The model was solved using a decomposed algorithm based on Benders' decomposition.The model and the algorithm were applied to a simple 3-node system and an actual 445-node system for verification,respectively.Test results show that the model can save 84.5 US $ cost for the testing three-node system,and the algorithm can solve the model for 445-node system within 5 min.The test results also illustrate that the proposed approach is efficient and suitable for large system calculation.

Optimal Consumption under Uncertainties: Random Horizon Stochastic Dynamic Roy’s Identity and Slutsky Equation

This paper extends Slutsky’s classic work on consumer theory to a random horizon stochastic dynamic framework in which the consumer has an inter-temporal planning horizon with uncertainties in future incomes and life span. Utility maximization leading to a set of ordinary wealth-dependent demand functions is performed. A dual problem is set up to derive the wealth compensated demand functions. This represents the first time that wealth-dependent ordinary demand functions and wealth compensated demand functions are obtained under these uncertainties. The corresponding Roy’s identity relationships and a set of random horizon stochastic dynamic Slutsky equations are then derived. The extension incorporates realistic characteristics in consumer theory and advances the conventional microeconomic study on consumption to a more realistic optimal control framework.

On stochastic optimal control of partially observable nonlinear quasi Hamiltonian systems

A stochastic optimal control strategy for partially observable nonlinear quasi Hamiltonian systems is proposed. The optimal control forces consist of two parts. The first part is determined by the conditions under which the stochastic optimal control problem of a partially observable nonlinear system is converted into that of a completely observable linear system. The second part is determined by solving the dynamical programming equation derived by applying the stochastic averaging method and stochastic dynamical programming principle to the completely observable linear control system. The response of the optimally controlled quasi Hamiltonian system is predicted by solving the averaged Fokker-Planck-Kolmogorov equation associated with the optimally controlled completely observable linear system and solving the Riccati equation for the estimated error of system states. An example is given to illustrate the procedure and effectiveness of the proposed control strategy.

A NEW STOCHASTIC OPTIMAL CONTROL STRATEGY FOR HYSTERETIC MR DAMPERS

A new stochastic optimal control strategy for randomly excited quasi-integrable Hamiltonian systems using magneto-rheological (MR) dampers is proposed. The dynamic be- havior of an MR damper is characterized by the Bouc-Wen hysteretic model. The control force produced by the MR damper is separated into a passive part incorporated in the uncontrolled system and a semi-active part to be determined. The system combining the Bouc-Wen hysteretic force is converted into an equivalent non-hysteretic nonlinear stochastic control system. Then It?o stochastic di?erential equations are derived from the equivalent system by using the stochastic averaging method. A dynamical programming equation for the controlled di?usion processes is established based on the stochastic dynamical programming principle. The non-clipping nonlin- ear optimal control law is obtained for a certain performance index by minimizing the dynamical programming equation. Finally, an example is given to illustrate the application and e?ectiveness of the proposed control strategy.

Neural-Network-Based Control for Discrete-Time Nonlinear Systems with Input Saturation Under Stochastic Communication Protocol

In this paper,an adaptive dynamic programming(ADP)strategy is investigated for discrete-time nonlinear systems with unknown nonlinear dynamics subject to input saturation.To save the communication resources between the controller and the actuators,stochastic communication protocols(SCPs)are adopted to schedule the control signal,and therefore the closed-loop system is essentially a protocol-induced switching system.A neural network(NN)-based identifier with a robust term is exploited for approximating the unknown nonlinear system,and a set of switch-based updating rules with an additional tunable parameter of NN weights are developed with the help of the gradient descent.By virtue of a novel Lyapunov function,a sufficient condition is proposed to achieve the stability of both system identification errors and the update dynamics of NN weights.Then,a value iterative ADP algorithm in an offline way is proposed to solve the optimal control of protocol-induced switching systems with saturation constraints,and the convergence is profoundly discussed in light of mathematical induction.Furthermore,an actor-critic NN scheme is developed to approximate the control law and the proposed performance index function in the framework of ADP,and the stability of the closed-loop system is analyzed in view of the Lyapunov theory.Finally,the numerical simulation results are presented to demonstrate the effectiveness of the proposed control scheme.

Optimal control strategies for stochastically excited quasi partially integrable Hamiltonian systems

In this paper two different control strategies designed to alleviate the response of quasi partially integrable Hamiltonian systems subjected to stochastic excitation are proposed. First, by using the stochastic averaging method for quasi partially integrable Hamiltonian systems, an n-DOF controlled quasi partially integrable Hamiltonian system with stochastic excitation is converted into a set of partially averaged It^↑o stochastic differential equations. Then, the dynamical programming equation associated with the partially averaged It^↑o equations is formulated by applying the stochastic dynamical programming principle. In the first control strategy, the optimal control law is derived from the dynamical programming equation and the control constraints without solving the dynamical programming equation. In the second control strategy, the optimal control law is obtained by solving the dynamical programming equation. Finally, both the responses of controlled and uncontrolled systems are predicted through solving the Fokker-Plank-Kolmogorov equation associated with fully averaged It^↑o equations. An example is worked out to illustrate the application and effectiveness of the two proposed control strategies.

Dynamic Programming for Multidimensional Stochastic Control Problems

In this paper we study a general multidimensional diffusion-type stochastic control problem. Our model contains the usual regular control problem,singular control problem and impulse control problem as special cases.Using a unified treatment of dynamic programming,we show that the value function of the problem is a viscosity solution of certain Hamilton-Jacobi-Bellman (HJB) quasi- variational inequality.The uniqueness of such a quasi-variational inequality is proved.

Neural-network-based stochastic linear quadratic optimal tracking control scheme for unknown discrete-time systems using adaptive dynamic programming

In this paper,a stochastic linear quadratic optimal tracking scheme is proposed for unknown linear discrete-time(DT)systems based on adaptive dynamic programming(ADP)algorithm.First,an augmented system composed of the original system and the command generator is constructed and then an augmented stochastic algebraic equation is derived based on the augmented system.Next,to obtain the optimal control strategy,the stochastic case is converted into the deterministic one by system transformation,and then an ADP algorithm is proposed with convergence analysis.For the purpose of realizing the ADP algorithm,three back propagation neural networks including model network,critic network and action network are devised to guarantee unknown system model,optimal value function and optimal control strategy,respectively.Finally,the obtained optimal control strategy is applied to the original stochastic system,and two simulations are provided to demonstrate the effectiveness of the proposed algorithm.

STOCHASTIC OPTIMAL CONTROL FOR THE RESPONSE OF QUASI NON-INTEGRABLE HAMILTONIAN SYSTEMS

A strategy is proposed based on the stochastic averaging method for quasi non- integrable Hamiltonian systems and the stochastic dynamical programming principle.The pro- posed strategy can be used to design nonlinear stochastic optimal control to minimize the response of quasi non-integrable Hamiltonian systems subject to Gaussian white noise excitation.By using the stochastic averaging method for quasi non-integrable Hamiltonian systems the equations of motion of a controlled quasi non-integrable Hamiltonian system is reduced to a one-dimensional av- eraged It stochastic differential equation.By using the stochastic dynamical programming princi- ple the dynamical programming equation for minimizing the response of the system is formulated. The optimal control law is derived from the dynamical programming equation and the bounded control constraints.The response of optimally controlled systems is predicted through solving the FPK equation associated with It stochastic differential equation.An example is worked out in detail to illustrate the application of the control strategy proposed.

Two-Stage Stochastic Dual Dynamic Programming for Transmission Expansion Planning with Significant Renewable Generation and N-k Criterion

The large-scale integration of renewable energy sources(RES)is the global trend to deal with the energy crisis and greenhouse emissions.Due to the intermittent nature of RES together with the uncertainty of load demand,the problem of transmission expansion planning(TEP)is facing more and more challenges from uncertainties.In this paper,the TEP problem is modeled as a two-stage formulation,so as to minimize the total of investment costs and generation costs.To ensure the utilization level of the RES generation,the expansion plan is required to provide sufficient transmission capacity for the integration of RES.Also,N-k security criterion is considered into the model,so the expansion plan can meet the required security criteria.The stochastic dual dynamic programming(SDDP)approach is applied to consider the uncertainties,and the whole model is solved by Benders’decomposition technique.Two case studies are carried out to compare the performance of the SDDP approach and the deterministic approach.Results show that the expansion plan obtained by the SDDP approach has a better performance than that of the deterministic approach.

ON THE CONVERGENCE OF SAMPLING ALGORITHMS FOR SOWING DYNAMIC STOCHASTIC PROGRAMMING

After describing a general sampling discretization algorithm for multistage continuous stochastic programming problems, we prove the global convergence of the al- gorithm under suitable conditions. The convergence of most available algorithms as well as new algorithms can thus be derived or improved as a special case of this general result.

Viability decision of linear discrete-time stochastic systems with probability criterion

In this paper, the optimal viability decision problem of linear discrete-time stochastic systems with probability criterion is investigated. Under the condition of sequence-reachable discrete-time dynamic systems, the existence theorem of optimal viability strategy is given and the solving procedure of the optimal strategy is provided based on dynamic programming. A numerical example shows the effectiveness of the proposed methods.

Nonlinear stochastic optimal bounded control of hysteretic systems with actuator saturation

A modified nonlinear stochastic optimal bounded control strategy for random excited hysteretic systems with actuator saturation is proposed. First, a controlled hysteretic system is converted into an equivalent nonlinear nonhysteretic stochastic system. Then, the partially averaged Itoe stochastic differential equation and dynamical programming equation are established, respectively, by using the stochastic averaging method for quasi non-integrable Hamiltonian systems and stochastic dynamical programming principle, from which the optimal control law consisting of optimal unbounded control and bang-bang control is derived. Finally, the response of optimally controlled system is predicted by solving the Fokker-Planck-Kolmogorov （FPK） equation associated with the fully averaged Itoe equation. Numerical results show that the proposed control strategy has high control effectiveness and efficiency.

Distributive Disturbance and Optimal Policy in Stochastic Control Model

To investigate the equilibrium relationships between the volatility of capital and income, taxation, and ance in a stochastic control model, the uniqueness of the solution to this model was proved by using the method of dynamic programming under the introduction of distributive disturbance and elastic labor supply. Furthermore, the effects of two types of shocks on labor-leisure choice, economic growth rate and welfare were numerically analyzed, and then the optimal tax policy was derived.