Policy Gradient Adaptive Dynamic Programming for Model-Free Multi-Objective Optimal Control

Dear Editor,In this letter,the multi-objective optimal control problem of nonlinear discrete-time systems is investigated.A data-driven policy gradient algorithm is proposed in which the action-state value function is used to evaluate the policy.In the policy improvement process,the policy gradient based method is employed.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems,in this paper,a new iterative adaptive dynamic programming algorithm,which is the discrete-time time-varying policy iteration(DTTV)algorithm,is developed.The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance.The admissibility of the iterative control law is analyzed.The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution.To implement the algorithm,neural networks are employed and a new implementation structure is established,which avoids solving the generalized Bellman equation in each iteration.Finally,the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm,where the mass and pendulum bar length are permitted to be time-varying parameters.The effectiveness of the developed method is illustrated by numerical results and comparisons.

THE OPTIMAL DEDUCTIBLE AND COVERAGE IN INSURANCE CONTRACTS AND EQUILIBRIUM RISK SHARING POLICIES

In this paper, we consider the optimal risk sharing problem between two parties in the insurance business: the insurer and the insured. The risk is allocated between the insurer and the insured by setting a deductible and coverage in the insurance contract. We obtain the optimal deductible and coverage by considering the expected product of the two parties' utilities of terminal wealth according to stochastic optimal control theory. An equilibrium policy is also derived for when there are both a deductible and coverage;this is done by modelling the problem as a stochastic game in a continuous-time framework. A numerical example is provided to illustrate the results of the paper.

Optimal quasi-periodic maintenance policies for two-unit series system

To investigate the effects of various random factors on the preventive maintenance （PM） decision-making of one type of two-unit series system, an optimal quasi-periodic PM policy is introduced. Assume that PM is perfect for unit 1 and only mechanical service for unit 2 in the model. PM activity is randomly performed according to a dynamic PM plan distributed in each implementation period. A replacement is determined based on the competing results of unplanned and planned replacements. The unplanned replacement is trigged by a catastrophic failure of unit 2, and the planned replacement is executed when the PM number reaches the threshold N. Through modeling and analysis, a solution algorithm for an optimal implementation period and the PM number is given, and optimal process and parametric sensitivity are provided by a numerical example. Results show that the implementation period should be decreased as soon as possible under the condition of meeting the needs of practice, which can increase mean operating time and decrease the long-run cost rate.

OPTIMAL HARVESTING POLICY FOR INSHORE-OFFSHORE FISHERY MODEL WITH IMPULSIVE DIFFUSION

This article studies the inshore-offshore fishery model with impulsive diffusion. The existence and global asymptotic stability of both the trivial periodic solution and the positive periodic solution are obtained. The complexity of this system is also analyzed. Moreover, the optimal harvesting policy are given for the inshore subpopulation, which includes the maximum sustainable yield and the corresponding harvesting effort.

Optimal switching policy for performance enhancement of distributed parameter systems based on event-driven control

This paper aims to improve the performance of a class of distributed parameter systems for the optimal switching of actuators and controllers based on event-driven control. It is assumed that in the available multiple actuators, only one actuator can receive the control signal and be activated over an unfixed time interval, and the other actuators keep dormant. After incorporating a state observer into the event generator, the event-driven control loop and the minimum inter-event time are ultimately bounded. Based on the event-driven state feedback control, the time intervals of unfixed length can be obtained. The optimal switching policy is based on finite horizon linear quadratic optimal control at the beginning of each time subinterval. A simulation example demonstrate the effectiveness of the proposed policy.

RECURSIVE UTILITY,PRODUCTIVE GOVERNMENT EXPENDITURE AND OPTIMAL FISCAL POLICY

This paper employs a stochastic endogenous growth model extended to the case of a recursive utility function which can disentangle intertemporal substitution from risk aversion to analyze productive government expenditure and optimal fiscal policy, particularly stresses the importance of factor income. First, the explicit solutions of the central planner＇s stochastic optimization problem are derived, the growth maximizing and welfare-maximizing government expenditure policies are obtained and their standing in conflict or coincidence depends upon intertemporal substitution. Second, the explicit solutions of the representative individual＇s stochastic optimization problem which permits to tax on capital income and labor income separately are derived ,and it is found that the effect of risk on growth crucially depends on the degree of risk aversion,the intertemporal elasticity of substitution and the capital income share. Finally, a flexible optimal tax policy which can be internally adjusted to a certain extent is derived, and it is found that the distribution of factor income plays an important role in designing the optimal tax policy.

Analysis of a POMDP Model for an Optimal Maintenance Problem with Multiple Imperfect Repairs

I consider a system whose deterioration follows a discrete-time and discrete-state Markov chain with an absorbing state. When the system is put into practice, I may select operation (wait), imperfect repair, or replacement at each discrete-time point. The true state of the system is not known when it is operated. Instead, the system is monitored after operation and some incomplete information concerned with the deterioration is obtained for decision making. Since there are multiple imperfect repairs, I can select one option from them when the imperfect repair is preferable to operation and replacement. To express this situation, I propose a POMDP model and theoretically investigate the structure of an optimal maintenance policy minimizing a total expected discounted cost for an unbounded horizon. Then two stochastic orders are used for the analysis of our problem.

Wind Turbine Optimal Preventive Maintenance Scheduling Using Fibonacci Search and Genetic Algorithm

Maintenance scheduling is essential and crucial for wind turbines (WTs) to avoid breakdowns andreduce maintenance costs. Many maintenance models have been developed for WTs’ maintenance planning, suchas corrective, preventive, and predictive maintenance. Due to communities’ dependence on WTs for electricityneeds, preventive maintenance is the most widely used method for maintenance scheduling. The downside tousing this approach is that preventive maintenance (PM) is often done in fixed intervals, which is inefficient. In thispaper, a more detailed maintenance plan for a 2 MW WT has been developed. The paper’s focus is to minimize aWT’s maintenance cost based on a WT’s reliability model. This study uses a two-layer optimization framework:Fibonacci and genetic algorithm. The first layer in the optimization method (Fibonacci) finds the optimal numberof PM required for the system. In the second layer, the optimal times for preventative maintenance and optimalcomponents to maintain have been determined to minimize maintenance costs. The Monte Carlo simulationestimates WT component failure times using their lifetime distributions from the reliability model. The estimatedfailure times are then used to determine the overall corrective and PM costs during the system’s lifetime. Finally,an optimal PM schedule is proposed for a 2 MW WT using the presented method. The method used in this papercan be expanded to a wind farm or similar engineering systems.

Distributive Disturbance and Optimal Policy in Stochastic Control Model

To investigate the equilibrium relationships between the volatility of capital and income, taxation, and ance in a stochastic control model, the uniqueness of the solution to this model was proved by using the method of dynamic programming under the introduction of distributive disturbance and elastic labor supply. Furthermore, the effects of two types of shocks on labor-leisure choice, economic growth rate and welfare were numerically analyzed, and then the optimal tax policy was derived.

Optimal policy for controlling two-server queueing systems with jockeying

This paper studies the optimal policy for joint control of admission, routing, service, and jockeying in a queueing system consisting of two exponential servers in parallel.Jobs arrive according to a Poisson process.Upon each arrival, an admission/routing decision is made, and the accepted job is routed to one of the two servers with each being associated with a queue.After each service completion, the servers have an option of serving a job from its own queue, serving a jockeying job from another queue, or staying idle.The system performance is inclusive of the revenues from accepted jobs, the costs of holding jobs in queues, the service costs and the job jockeying costs.To maximize the total expected discounted return, we formulate a Markov decision process(MDP) model for this system.The value iteration method is employed to characterize the optimal policy as a hedging point policy.Numerical studies verify the structure of the hedging point policy which is convenient for implementing control actions in practice.

Optimal Water Pipe Replacement Policy

Water scarcity is the major problem confronting both urban and rural dwellers in Enugu State. This scarcity emanated from indiscriminate pipe failure, lack of adequate maintenance, uncertainty on the time of repair or replacement of pipes etc. There is no systematic approach to determining replacement or repair time of the pipes. Hence, the rule of thumb is used in making such a vital decision. The population is increasing, houses are built but the network is not expanded and the existing ones that were installed for no less than two to three decades ago are not maintained. These compounded the problem of scarcity of water in the state. Replacement or repair of water pipes when they are seen spilling water cannot solve this lingering problem. The solution can be achieved by developing an adequate predictive model for water pipe replacement. Hence, this research is aimed at providing a solution to this problem of water scarcity by suggesting a policy that will be used for better planning. The interests in this paper were to obtain a water pipe failure model, the intensity function λ(t) [failure rate], the reliability R(t) and the optimal time of replacement and they were achieved. It was observed that the failure rate of the pipes increases with time while their reliability deteriorates with time. Hence, the Optimal replacement policy is that each pipe should be replaced after 4th break when the reliability = 0.0011.

Optimal Price Strategy under Price-Matching Policy

The paper explores the optimal price strategy under the price-matching policy. First, the paper formulates the demand function under the price match policy and then discovers the retailer’s best response facing the price-matching pressure. From the theoretical analysis, we discover how the number of retailers plays an important role during the competition. When only two retailers are involved, the final prices may not converge to a single value. However, when more retailers are involved, the final price will converge to a single value. We also use numerical studies to illuminate the change of the prices over the time period, the sensitivity of the final price to the increment/decrement of initial prices. Finally, we provide managerial suggestions to both producers and retailers.

Optimal Static Partition Configuration in ARINC653 System

ARINC653 systems, which have been widely used in avionics industry, are an important class of safety-critical applications. Partitions are the core concept in the Arinc653 system architecture. Due to the existence of partitions, the system designer must allocate adequate time slots statically to each partition in the design phase. Although some time slot allocation policies could be borrowed from task scheduling policies, no existing literatures give an optimal allocation policy. In this paper, we present a partition configuration policy and prove that this policy is optimal in the sense that if this policy fails to configure adequate time slots to each partition, nor do other policies. Then, by simulation, we show the effects of different partition configuration policies on time slot allocation of partitions and task response time, respectively.

THE OPTIMAL STRATEGY FOR INSURANCE COMPANY UNDER THE INFLUENCE OF TERMINAL VALUE

This paper considers a model of an insurance company which is allowed to invest a risky asset and to purchase proportional reinsurance. The objective is to find the policy which maximizes the expected total discounted dividend pay-out until the time of bankruptcy and the terminal value of the company under liquidity constraint. We find the solution of this problem via solving the problem with zero terminal value. We also analyze the influence of terminal value on the optimal policy.

Replenishment policy and inventory optimization for supply-hub with liability period consideration

A replenishment decision-making model for supply-hub is firstly established from the angle of supplier, and optimal replenishment decision of the supplier is analyzed. Then, inventory optimization model for supply-hub is formulated from the angle of the manufacturer, and the optimization algorithm for obtaining optimal inventory levels is given. The result shows that liability period decides the share of the inventory cost between two sides in supply chain. With the increase of liability period, the service level has been quickly reduced even though the manufacturer's cost has been cut down by transferring the inventory cost to the supplier. As to the safety inventory, if the lower bound of components safety inventory increases, the supplier's cost will rise up more slowly than the liability period does, while the service levels increases as the safety inventory's lower bound is raised.

Optimization on bicriterion policies for M/G/1 system with second optional service

We compare the optimal operating cost of the two bicriterion policies, <p,T> and <p,N>, for an M/G/1 queueing system with second optional service, in which the length of the vacation period is randomly controlled either by the number of arrivals during the idle period or by a timer. After all the customers are served in the queue exhaustively, the server immediately takes a vacation and may operate <p,T> policy or <p,N> policy. For the two bicriterion policies, the total average cost function per unit time is developed to search the optimal stationary operating policies at a minimum cost. Based upon the optimal cost the explicit forms for joint optimum threshold values of (p,T) and (p,N) are obtained.

STUDY ON THE OPTIMIZATION OF TRANSPORT CONTROL POLICY IN COMMUNICATION NETWORK

In communication networks with policy-based Transport Control on-Demand (TCoD) function,the transport control policies play a great impact on the network effectiveness. To evaluate and optimize the transport policies in communication network,a policy-based TCoD network model is given and a comprehensive evaluation index system of the network effectiveness is put forward from both network application and handling mechanism perspectives. A TCoD network prototype system based on Asynchronous Transfer Mode/Multi-Protocol Label Switching (ATM/MPLS) is introduced and some experiments are performed on it. The prototype system is evaluated and analyzed with the comprehensive evaluation index system. The results show that the index system can be used to judge whether the communication network can meet the application requirements or not,and can provide references for the optimization of the transport policies so as to improve the communication network effectiveness.

Optimization and Regulation Policy for Land Use Changes Based on Low-carbon Emission in Developed Regions of China

Land use and cover change(LUCC) is one of the important causes of the Earth’s carbon cycle imbalances resulting from failure in optimizing land use. The solution to this problem has been the hotspot of research in land and environmental science. We took 'low carbon', 'energy saving' and 'high-efficiency' as the goals of land use optimization,and integrated Markov-CA(Cellular Automaton),the Grid-Fractal model and GIS,in order to study carbon emission objective function,to establish a simulation method for land use spatial allocation optimization,to evaluate the effect of the method on carbon emissions. Regulation policy on three types of land use spatial allocation was proposed,including 'low-carbon type', 'low-carbon-economic type' and 'economic type'. We applied the method to analyze the land use spatial allocation in Taixing City of the 'Yangtze River Delta' regions in China,and obtained the following results:(i) The three optimization types would improve carbon emissions by 3. 21%,1. 80% and 0. 36% respectively in 2020,compared with 2010;(ii) The actual planning for 2020 was close to the 'low-carbon-economic type';(iii) The optimization method and regulation policy,combining local optimization and global control,could meet the sustainable multi-objective requirements for low-carbon constraints of land use spatial allocation. The result of this research could also serve as a reference for exploration into patterns of regional low-carbon land use and measures for energy saving and emission reduction.

Distributionally Robust Optimal Dispatch of Virtual Power Plant Based on Moment of Renewable Energy Resource

Virtual power plants can effectively integrate different types of distributed energy resources,which have become a new operation mode with substantial advantages such as high flexibility,adaptability,and economy.This paper proposes a distributionally robust optimal dispatch approach for virtual power plants to determine an optimal day-ahead dispatch under uncertainties of renewable energy sources.The proposed distributionally robust approach characterizes probability distributions of renewable power output by moments.In this regard,the faults of stochastic optimization and traditional robust optimization can be overcome.Firstly,a second-order cone-based ambiguity set that incorporates the first and second moments of renewable power output is constructed,and a day-ahead two-stage distributionally robust optimization model is proposed for virtual power plants participating in day-ahead electricity markets.Then,an effective solution method based on the affine policy and second-order cone duality theory is employed to reformulate the proposed model into a deterministic mixed-integer second-order cone programming problem,which improves the computational efficiency of the model.Finally,the numerical results demonstrate that the proposed method achieves a better balance between robustness and economy.They also validate that the dispatch strategy of virtual power plants can be adjusted to reduce costs according to the moment information of renewable power output.