Policy Gradient Adaptive Dynamic Programming for Model-Free Multi-Objective Optimal Control

Dear Editor,In this letter,the multi-objective optimal control problem of nonlinear discrete-time systems is investigated.A data-driven policy gradient algorithm is proposed in which the action-state value function is used to evaluate the policy.In the policy improvement process,the policy gradient based method is employed.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems,in this paper,a new iterative adaptive dynamic programming algorithm,which is the discrete-time time-varying policy iteration(DTTV)algorithm,is developed.The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance.The admissibility of the iterative control law is analyzed.The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution.To implement the algorithm,neural networks are employed and a new implementation structure is established,which avoids solving the generalized Bellman equation in each iteration.Finally,the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm,where the mass and pendulum bar length are permitted to be time-varying parameters.The effectiveness of the developed method is illustrated by numerical results and comparisons.

Replenishment policy and inventory optimization for supply-hub with liability period consideration

A replenishment decision-making model for supply-hub is firstly established from the angle of supplier, and optimal replenishment decision of the supplier is analyzed. Then, inventory optimization model for supply-hub is formulated from the angle of the manufacturer, and the optimization algorithm for obtaining optimal inventory levels is given. The result shows that liability period decides the share of the inventory cost between two sides in supply chain. With the increase of liability period, the service level has been quickly reduced even though the manufacturer's cost has been cut down by transferring the inventory cost to the supplier. As to the safety inventory, if the lower bound of components safety inventory increases, the supplier's cost will rise up more slowly than the liability period does, while the service levels increases as the safety inventory's lower bound is raised.

THE OPTIMAL DEDUCTIBLE AND COVERAGE IN INSURANCE CONTRACTS AND EQUILIBRIUM RISK SHARING POLICIES

In this paper, we consider the optimal risk sharing problem between two parties in the insurance business: the insurer and the insured. The risk is allocated between the insurer and the insured by setting a deductible and coverage in the insurance contract. We obtain the optimal deductible and coverage by considering the expected product of the two parties' utilities of terminal wealth according to stochastic optimal control theory. An equilibrium policy is also derived for when there are both a deductible and coverage;this is done by modelling the problem as a stochastic game in a continuous-time framework. A numerical example is provided to illustrate the results of the paper.

Optimization on bicriterion policies for M/G/1 system with second optional service

We compare the optimal operating cost of the two bicriterion policies, <p,T> and <p,N>, for an M/G/1 queueing system with second optional service, in which the length of the vacation period is randomly controlled either by the number of arrivals during the idle period or by a timer. After all the customers are served in the queue exhaustively, the server immediately takes a vacation and may operate <p,T> policy or <p,N> policy. For the two bicriterion policies, the total average cost function per unit time is developed to search the optimal stationary operating policies at a minimum cost. Based upon the optimal cost the explicit forms for joint optimum threshold values of (p,T) and (p,N) are obtained.

Construction of Legal System of China's Farmland Protection under the Coexistence of Multiple Objectives:Historical Logic,Practical Problems and Optimization Paths

[Objectives]To explore the evolution of the legal system of farmland protection and explore the rules and characteristics of policy development based on the theory and logic of institutional change since China's reform and opening up,reveal the problems and deep-seated reasons of its legislation,clarify the direction of farmland protection in the new period,and solve the"non-agricultural""non-grain"and ecological problems of farmland.[Methods]Literature analysis and inductive deduction methods were used.[Results]The evolution of the farmland protection legal system has gone through the process of"national consciousness-policy guidelines-institutional system",the change from"single subject to multiple subjects";change from the use of"one-way administrative means to coordinated use of administrative,economic and technical means".The practical problems of the farmland protection legal system are mainly due to the insufficient systematization of the farmland protection legal system itself,the generalization of quantity protection,the transformation of quality protection,and the absence of ecological protection.[Conclusions]It is recommended to improve the existing farmland protection legal system from the establishment of the Farmland Protection Law,the improvement of the farmland protection public participation mechanism and supervision mechanism,the establishment of the farmland quality construction and improvement system,the differentiated farmland occupation and supplementation balance system,and the ecological restoration system.

STUDY ON THE OPTIMIZATION OF TRANSPORT CONTROL POLICY IN COMMUNICATION NETWORK

In communication networks with policy-based Transport Control on-Demand (TCoD) function,the transport control policies play a great impact on the network effectiveness. To evaluate and optimize the transport policies in communication network,a policy-based TCoD network model is given and a comprehensive evaluation index system of the network effectiveness is put forward from both network application and handling mechanism perspectives. A TCoD network prototype system based on Asynchronous Transfer Mode/Multi-Protocol Label Switching (ATM/MPLS) is introduced and some experiments are performed on it. The prototype system is evaluated and analyzed with the comprehensive evaluation index system. The results show that the index system can be used to judge whether the communication network can meet the application requirements or not,and can provide references for the optimization of the transport policies so as to improve the communication network effectiveness.

Optimization and Regulation Policy for Land Use Changes Based on Low-carbon Emission in Developed Regions of China

Land use and cover change(LUCC) is one of the important causes of the Earth’s carbon cycle imbalances resulting from failure in optimizing land use. The solution to this problem has been the hotspot of research in land and environmental science. We took 'low carbon', 'energy saving' and 'high-efficiency' as the goals of land use optimization,and integrated Markov-CA(Cellular Automaton),the Grid-Fractal model and GIS,in order to study carbon emission objective function,to establish a simulation method for land use spatial allocation optimization,to evaluate the effect of the method on carbon emissions. Regulation policy on three types of land use spatial allocation was proposed,including 'low-carbon type', 'low-carbon-economic type' and 'economic type'. We applied the method to analyze the land use spatial allocation in Taixing City of the 'Yangtze River Delta' regions in China,and obtained the following results:(i) The three optimization types would improve carbon emissions by 3. 21%,1. 80% and 0. 36% respectively in 2020,compared with 2010;(ii) The actual planning for 2020 was close to the 'low-carbon-economic type';(iii) The optimization method and regulation policy,combining local optimization and global control,could meet the sustainable multi-objective requirements for low-carbon constraints of land use spatial allocation. The result of this research could also serve as a reference for exploration into patterns of regional low-carbon land use and measures for energy saving and emission reduction.

Optimization on Water Resource System Operation Policy during Drought

The mixed linear programming model is commonly recognized to be an effective means for searching optimal reservoir operation policy in water resources system. In this paper a multi-objective mixed integer linear programming model is set up to obtain the optimal operation policy of multi-reservoir water supply system during drought, which is able to consider the operation rule of reservoir-group system within longer-term successive drought periods, according to the basic connotation of indexes expressing the water-supply risk of reservoir during drought, that is, reliability, resilience and vulnerability of reservoir water supply, and mathematical programming principles. The model-solving procedures, particularly, the decomposition-adjustment algorithm, are proposed based on characteristics of the model structure. The principle of model-solving technique is to decompose the complex system into several smaller sub-systems on which some ease-solving mathematical models may be established. The objective of this optimization model aims at maximizing the reliability of water supply and minimizing the maximum water-shortage of single time-period within water- supply system during drought. The multi-objective mixed integer linear programming model and proposed solving procedures are applied to a case study of reservoir-group water-supply system in Huanghe-Huaihe River Basin, China. The desired water-shortage distribution within the system operation term and the maximum shortage of single time-period are achieved. The results of case study verifies that the lighter water-shortage distributed evenly among several time-periods can avoid the calamities resulted from severe water shortage concentrated on a few time-periods during drought.

Policy Optimization Study Based on Evolutionary Learning

In order to achieve an intelligent and automated self-management network,dynamic policy configuration and selection are needed.A certain policy only suits to a certain network environment.If the network environment changes,the certain policy does not suit any more.Thereby,the policy-based management should also have similar "natural selection" process.Useful policy will be retained,and policies which have lost their effectiveness are eliminated.A policy optimization method based on evolutionary learning was proposed.For different shooting times,the priority of policy with high shooting times is improved,while policy with a low rate has lower priority,and long-term no shooting policy will be dormant.Thus the strategy for the survival of the fittest is realized,and the degree of self-learning in policy management is improved.

Analysis on the Development Trend of Contemporary Local Governance Mode from the perspective of Economic Policy and Environmental Capacity Optimization

As the basic level of national governance, local governance is deeply influenced by administrative tradition, political system and economic development level of a country. With more than thirty years of reform and opening up, the pattern of local governance has become diversified. The diversity of governance model cannot be managed in the same way, however, it cannot affect the government’s ability to play the initiative role of governance. We need to innovate and improve the existing local governance models.

Optimizing the Multi-Objective Discrete Particle Swarm Optimization Algorithm by Deep Deterministic Policy Gradient Algorithm

Deep deterministic policy gradient(DDPG)has been proved to be effective in optimizing particle swarm optimization(PSO),but whether DDPG can optimize multi-objective discrete particle swarm optimization(MODPSO)remains to be determined.The present work aims to probe into this topic.Experiments showed that the DDPG can not only quickly improve the convergence speed of MODPSO,but also overcome the problem of local optimal solution that MODPSO may suffer.The research findings are of great significance for the theoretical research and application of MODPSO.

Optimal quasi-periodic maintenance policies for two-unit series system

To investigate the effects of various random factors on the preventive maintenance （PM） decision-making of one type of two-unit series system, an optimal quasi-periodic PM policy is introduced. Assume that PM is perfect for unit 1 and only mechanical service for unit 2 in the model. PM activity is randomly performed according to a dynamic PM plan distributed in each implementation period. A replacement is determined based on the competing results of unplanned and planned replacements. The unplanned replacement is trigged by a catastrophic failure of unit 2, and the planned replacement is executed when the PM number reaches the threshold N. Through modeling and analysis, a solution algorithm for an optimal implementation period and the PM number is given, and optimal process and parametric sensitivity are provided by a numerical example. Results show that the implementation period should be decreased as soon as possible under the condition of meeting the needs of practice, which can increase mean operating time and decrease the long-run cost rate.

OPTIMAL HARVESTING POLICY FOR INSHORE-OFFSHORE FISHERY MODEL WITH IMPULSIVE DIFFUSION

This article studies the inshore-offshore fishery model with impulsive diffusion. The existence and global asymptotic stability of both the trivial periodic solution and the positive periodic solution are obtained. The complexity of this system is also analyzed. Moreover, the optimal harvesting policy are given for the inshore subpopulation, which includes the maximum sustainable yield and the corresponding harvesting effort.

Distributed optimization of electricity-Gas-Heat integrated energy system with multi-agent deep reinforcement learning

The coordinated optimization problem of the electricity-gas-heat integrated energy system(IES)has the characteristics of strong coupling,non-convexity,and nonlinearity.The centralized optimization method has a high cost of communication and complex modeling.Meanwhile,the traditional numerical iterative solution cannot deal with uncertainty and solution efficiency,which is difficult to apply online.For the coordinated optimization problem of the electricity-gas-heat IES in this study,we constructed a model for the distributed IES with a dynamic distribution factor and transformed the centralized optimization problem into a distributed optimization problem in the multi-agent reinforcement learning environment using multi-agent deep deterministic policy gradient.Introducing the dynamic distribution factor allows the system to consider the impact of changes in real-time supply and demand on system optimization,dynamically coordinating different energy sources for complementary utilization and effectively improving the system economy.Compared with centralized optimization,the distributed model with multiple decision centers can achieve similar results while easing the pressure on system communication.The proposed method considers the dual uncertainty of renewable energy and load in the training.Compared with the traditional iterative solution method,it can better cope with uncertainty and realize real-time decision making of the system,which is conducive to the online application.Finally,we verify the effectiveness of the proposed method using an example of an IES coupled with three energy hub agents.

Optimal switching policy for performance enhancement of distributed parameter systems based on event-driven control

This paper aims to improve the performance of a class of distributed parameter systems for the optimal switching of actuators and controllers based on event-driven control. It is assumed that in the available multiple actuators, only one actuator can receive the control signal and be activated over an unfixed time interval, and the other actuators keep dormant. After incorporating a state observer into the event generator, the event-driven control loop and the minimum inter-event time are ultimately bounded. Based on the event-driven state feedback control, the time intervals of unfixed length can be obtained. The optimal switching policy is based on finite horizon linear quadratic optimal control at the beginning of each time subinterval. A simulation example demonstrate the effectiveness of the proposed policy.

RECURSIVE UTILITY,PRODUCTIVE GOVERNMENT EXPENDITURE AND OPTIMAL FISCAL POLICY

This paper employs a stochastic endogenous growth model extended to the case of a recursive utility function which can disentangle intertemporal substitution from risk aversion to analyze productive government expenditure and optimal fiscal policy, particularly stresses the importance of factor income. First, the explicit solutions of the central planner＇s stochastic optimization problem are derived, the growth maximizing and welfare-maximizing government expenditure policies are obtained and their standing in conflict or coincidence depends upon intertemporal substitution. Second, the explicit solutions of the representative individual＇s stochastic optimization problem which permits to tax on capital income and labor income separately are derived ,and it is found that the effect of risk on growth crucially depends on the degree of risk aversion,the intertemporal elasticity of substitution and the capital income share. Finally, a flexible optimal tax policy which can be internally adjusted to a certain extent is derived, and it is found that the distribution of factor income plays an important role in designing the optimal tax policy.

Structural Topology Optimization by Combining BESO with Reinforcement Learning

In this paper,a new algorithm combining the features of bi-direction evolutionary structural optimization(BESO)and reinforcement learning(RL)is proposed for continuum structural topology optimization(STO).In contrast to conventional approaches which only generate a certain quasi-optimal solution,the goal of the combined method is to provide more quasi-optimal solutions for designers such as the idea of generative design.Two key components were adopted.First,besides sensitivity,value function updated by Monte-Carlo reinforcement learning was utilized to measure the importance of each element,which made the solving process convergent and closer to the optimum.Second,ε-greedy policy added a random perturbation to the main search direction so as to extend the search ability.Finally,the quality and diversity of solutions could be guaranteed by controlling the value of compliance as well as Intersection-over-Union(IoU).Results of several 2D and 3D compliance minimization problems,including a geometrically nonlinear case,show that the combined method is capable of generating a group of good and different solutions that satisfy various possible requirements in engineering design within acceptable computation cost.

Optimization Model of Cattle Husbandry for Rural Household in Cambodia

Cambodia is one of the Southeast Asia. With the agricultural market integration, Cambodia rural household is adjusting livestock structure naturally. In order to provide suitable support for agriculture policy, the authors conducted a survey on 204 rural household in Cambodia. This article uses the optimization model, considering rural labor, cattle size, and animal disease risk, to analyze and get optimum result range. The result shows that the more off-farm job opportunity, suitable cattle feed structure, and investment on public health for cattle, the household income in rural Cambodia will increase.

An Integrated Water Resources and Economic Approach for Optimizing Water Allocation Policies

Reservoir/river systems analysis models are generally used in the formulation and evaluation of alternative plans for responding to water related problems and needs. One of the main problems is the water resources allocation and the cost associated with pumping, if needed. Taking the appropriate decision is considered as a techno-economic issue. The case study presented in this paper involves a complex system of three dams, two pumping stations and two diversion structures all serving an agricultural production unit. The objective of this research is to determine a suitable and feasible water allocation/pumping policy as a “trade-off” between minimizing the water deficiency and the cost of pumping. To achieve this objective, a water resources model was developed using HEC-5. A multi-criteria decision approach was implemented to determine the most appropriate water release policy and the capacity of the water diversion facilities. The parameters used were subject to a sensitivity analysis to assess their relative impact on the determined policy. The suggested release policy allows a reduction of half the total of the pumping costs with only 3% reduction in the water allocation reliability, as measured by the failure frequency of demand satisfaction and the average shortage index.