Policy Gradient Adaptive Dynamic Programming for Model-Free Multi-Objective Optimal Control

Dear Editor,In this letter,the multi-objective optimal control problem of nonlinear discrete-time systems is investigated.A data-driven policy gradient algorithm is proposed in which the action-state value function is used to evaluate the policy.In the policy improvement process,the policy gradient based method is employed.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems,in this paper,a new iterative adaptive dynamic programming algorithm,which is the discrete-time time-varying policy iteration(DTTV)algorithm,is developed.The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance.The admissibility of the iterative control law is analyzed.The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution.To implement the algorithm,neural networks are employed and a new implementation structure is established,which avoids solving the generalized Bellman equation in each iteration.Finally,the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm,where the mass and pendulum bar length are permitted to be time-varying parameters.The effectiveness of the developed method is illustrated by numerical results and comparisons.

Optimization of Gas Production from Hydrate-Bearing Sediments with Fluctuation Characteristics

As an important source of low-carbon,clean fossil energy,natural gas hydrate plays an important role in improving the global energy consumption structure.Developing the hydrate industry in the South China Sea is important to achieving‘carbon peak and carbon neutrality’goals as soon as possible.Deep-water areas subjected to the action of long-term stress and tectonic movement have developed complex and volatile terrains,and as such,the morphologies of hydrate-bearing sediments(HBSs)fluctuate correspondingly.The key to numerically simulating HBS morphologies is the establishment of the conceptual model,which represents the objective and real description of the actual geological body.However,current numerical simulation models have characterized HBSs into horizontal strata without considering the fluctuation characteristics.Simply representing the HBS as a horizontal element reduces simulation accuracy.Therefore,the commonly used horizontal HBS model and a model considering the HBS’s fluctuation characteristics with the data of the SH2 site in the Shenhu Sea area were first constructed in this paper.Then,their production behaviors were compared,and the huge impact of the fluctuation characteristics on HBS production was determined.On this basis,the key parameters affecting the depressurization production of the fluctuating HBSs were studied and optimized.The research results show that the fluctuation characteristics have an obvious influence on the hydrate production of HBSs by affecting their temperatures and pressure distributions,as well as the transmission of the pressure drop and methane gas discharge.Furthermore,the results show that the gas productivity of fluctuating HBSs was about 5%less than that of horizontal HBSs.By optimizing the depressurization amplitude,well length,and layout location of vertical wells,the productivity of fluctuating HBSs increased by about 56.6%.

Multi-surrogate framework with an adaptive selection mechanism for production optimization

Data-driven surrogate models that assist with efficient evolutionary algorithms to find the optimal development scheme have been widely used to solve reservoir production optimization problems.However,existing research suggests that the effectiveness of a surrogate model can vary depending on the complexity of the design problem.A surrogate model that has demonstrated success in one scenario may not perform as well in others.In the absence of prior knowledge,finding a promising surrogate model that performs well for an unknown reservoir is challenging.Moreover,the optimization process often relies on a single evolutionary algorithm,which can yield varying results across different cases.To address these limitations,this paper introduces a novel approach called the multi-surrogate framework with an adaptive selection mechanism(MSFASM)to tackle production optimization problems.MSFASM consists of two stages.In the first stage,a reduced-dimensional broad learning system(BLS)is used to adaptively select the evolutionary algorithm with the best performance during the current optimization period.In the second stage,the multi-objective algorithm,non-dominated sorting genetic algorithm II(NSGA-II),is used as an optimizer to find a set of Pareto solutions with good performance on multiple surrogate models.A novel optimal point criterion is utilized in this stage to select the Pareto solutions,thereby obtaining the desired development schemes without increasing the computational load of the numerical simulator.The two stages are combined using sequential transfer learning.From the two most important perspectives of an evolutionary algorithm and a surrogate model,the proposed method improves adaptability to optimization problems of various reservoir types.To verify the effectiveness of the proposed method,four 100-dimensional benchmark functions and two reservoir models are tested,and the results are compared with those obtained by six other surrogate-model-based methods.The results demonstrate that our approach can obtain the maximum net present value(NPV)of the target production optimization problems.

Optimal production lot size with process deterioration under an extended inspection policy

A mathematical model to determine the optimal production lot size for a deteriorating production system under an extended product inspection policy is developed. The last-K product inspection policy is considered so that the nonconforming items can be reduced, under which the last K products in a production lot are inspected and the nonconforming items from those inspected are reworked. Consider that the products produced towards the end of a production lot are more likely to be nonconforming, is proposed an extended product inspection policy for a deteriorating production system. That is, in a production lot, product inspections are performed among the middle K1 items and after inspections, all of the last K2 products are directly reworked without inspections. Our objective here is the joint optimization of the production lot size and the corresponding extended inspection policy such that the expected total cost per unit time is minimized. Since there is no closed form expression for our optimal policy, the existence for the optimal production inspection policy and an upper bound for the optimal lot size are obtained. Furthermore, an efficient solution procedure is provided to search for the optimal policy. Finally, numerical examples are given to illustrate the proposed model and indicate that the expected total cost per unit time of our product inspection model is less than that of the last-K inspection policy.

OPTIMAL HARVESTING POLICY FOR INSHORE-OFFSHORE FISHERY MODEL WITH IMPULSIVE DIFFUSION

This article studies the inshore-offshore fishery model with impulsive diffusion. The existence and global asymptotic stability of both the trivial periodic solution and the positive periodic solution are obtained. The complexity of this system is also analyzed. Moreover, the optimal harvesting policy are given for the inshore subpopulation, which includes the maximum sustainable yield and the corresponding harvesting effort.

RECURSIVE UTILITY,PRODUCTIVE GOVERNMENT EXPENDITURE AND OPTIMAL FISCAL POLICY

This paper employs a stochastic endogenous growth model extended to the case of a recursive utility function which can disentangle intertemporal substitution from risk aversion to analyze productive government expenditure and optimal fiscal policy, particularly stresses the importance of factor income. First, the explicit solutions of the central planner＇s stochastic optimization problem are derived, the growth maximizing and welfare-maximizing government expenditure policies are obtained and their standing in conflict or coincidence depends upon intertemporal substitution. Second, the explicit solutions of the representative individual＇s stochastic optimization problem which permits to tax on capital income and labor income separately are derived ,and it is found that the effect of risk on growth crucially depends on the degree of risk aversion,the intertemporal elasticity of substitution and the capital income share. Finally, a flexible optimal tax policy which can be internally adjusted to a certain extent is derived, and it is found that the distribution of factor income plays an important role in designing the optimal tax policy.

Forecasting of China's natural gas production and its policy implications

With the vigorous promotion of energy conservation and implementation of clean energy strategies,China＇s natural gas industry has entered a rapid development phase,and natural gas is playing an increasingly important role in China＇s energy structure.This paper uses a Generalized Weng model to forecast Chinese regional natural gas production,where accuracy and reasonableness compared with other predictions are enhanced by taking remaining estimated recoverable resources as a criterion.The forecast shows that China＇s natural gas production will maintain a rapid growth with peak gas of 323 billion cubic meters a year coming in 2036;in 2020,natural gas production will surpass that of oil to become a more important source of energy.Natural gas will play an important role in optimizing China＇s energy consumption structure and will be a strategic replacement of oil.This will require that exploration and development of conventional natural gas is highly valued and its industrial development to be reasonably planned.As well,full use should be made of domestic and international markets.Initiative should also be taken in the exploration and development of unconventional and deepwater gas,which shall form a complement to the development of China＇s conventional natural gas industry.

Optimal switching policy for performance enhancement of distributed parameter systems based on event-driven control

This paper aims to improve the performance of a class of distributed parameter systems for the optimal switching of actuators and controllers based on event-driven control. It is assumed that in the available multiple actuators, only one actuator can receive the control signal and be activated over an unfixed time interval, and the other actuators keep dormant. After incorporating a state observer into the event generator, the event-driven control loop and the minimum inter-event time are ultimately bounded. Based on the event-driven state feedback control, the time intervals of unfixed length can be obtained. The optimal switching policy is based on finite horizon linear quadratic optimal control at the beginning of each time subinterval. A simulation example demonstrate the effectiveness of the proposed policy.

Modeling optimal oil production paths under risk service contracts

Due to the rigorous fiscal terms and huge potential risk of risk service contracts,optimizing oil production paths is one of the main challenges in designing oilfield development plans.In this paper,an oil production path optimization model is developed to maximize economic benefits within constraints of technology factors and oil contracts.This analysis describes the effects of risk service contract terms on parameters of inputs and outputs and quantifies the relationships between production and production time,revenues,investment and costs.An oil service development and production project is illustrated in which the optimal production path under its own geological conditions and contract terms is calculated.The influences of oil price,service fees per barrel and operating costs on the optimal production have been examined by sensitivity analysis.The results show that the oil price has the largest impact on the optimal production,which is negatively related to oil price and positively related to service fees per barrel and operating costs.

Exploiting Virtual Elasticity of Production Systems for Respecting OTD-Part 1: Post-Optimality Conditions for Ergodic Order Arrivals in Fixed Capacity Regimes

Respecting the on-time-delivery (OTD) for manufacturing orders is mandatory. This depends, however, on the probability distribution of incoming order rate. The case of non-equal distribution, such as aggregated arrivals, may compromise the observance of on-time supplies for some orders. The purpose of this paper is to evaluate the conditions of post-optimality for stochastic order rate governed production systems in order to observe OTD. Instead of a heuristic or a simulative exploration, a Cartesian-based approach is applied to developing the necessary and sufficient mathematical condition to solve the problem statement. The research result demonstrates that increasing speed of throughput reveals a latent capacity, which allows arrival orders above capacity limits to be backlog-buffered and rescheduled for OTD, exploiting the virtual manufacturing elasticity inherent to all production systems to increase OTD reliability of non JIT-based production systems.

International Conference on Sustainable Rice Production - Policy, Technology and Extension Celebration Activity for International Year of Rice and World Food Day 2004

Rice is the staple food of more than half of the world’s Population.The production,processing and management of paddy rice have provided the basic conditions for the living of mankind.The production of rice has not only created

Optimal production lot sizing model in a supply chain with periodically fixed demand considering learning effect

This paper presents an optimal production model for manufacturer in a supply chain with a fixed demand at a fixed interval with respect to the learning effect on production capacity. An algorithm is employed to find the optimal delay time for production and production time sequentially. It is found that the optimal delay time for production and the production time are not static, but dynamic and variant with time. It is important for a manufacturer to schedule the production so as to prevent facilities and workers from idling.

Study on Optimal Strategy of Grade Transition in Industrial Fluidized Bed Gas-Phase Polyethylene Production Process

A model of grade transition is presented for a commercialized fluidized bed gas-phase polyethylene production process.The quantity of off-specification product and the time of grade transition can be minimized by the optimization of operating variables,such as polymerization temperature,the ratio of hydrogen to ethylene,the ratio of co-monomer to ethylene,feed rate of catalyst,and bed level.A new performance index,the ratio of melt flow(MFR),is included in the objective function,for restraining the sharp adjustment of operation variables and narrowing the distribution of molecular weight of the resion.It is recommended that catalyst feed rate and bed level are decreased in order to reduce the grade transition time and the quantity of off-specification product.This optimization problem is solved by an algorithm of sequential quadratic programming (SQP) in MATLAB. There is considerable difference between the forward transition and reverse transition of grade with regard to the operating variables due to the non-linearity of the system.The grade transition model is extended to a high space time yield(STY)Process with the so-called condensed model operation.In the end,an optimization strategy for multi-product transition is proposed with two-level optimization of the objective function J(x,u) on the the basis of the optimal grade transition model.A sequential transition of six commercial polyethylene grades in illustrated for an optimal multi-product operation.

Rapid diagnostic method for transplutonium isotope production in high flux reactors

Transplutonium isotopes are scarce and need to be produced by irradiation in high flux reactors.However,their production is inefficient,and optimization studies are necessary.This study analyzes the physical nature of transplutonium isotope produc-tion using ^(252)Cf,^(244)Cm,^(242)Cm,and ^(238)Pu as examples.Traditional methods based on the Monte Carlo burnup calculation have the limitations of many calculations and cannot analyze the individual energy intervals in detail;thus,they cannot sup-port the refined evaluation,screening,and optimization of the irradiation schemes.After understanding the physical nature and simplifying the complexity of the production process,we propose a rapid diagnostic method for evaluating radiation schemes based on the concepts“single energy interval value(SEIV)”and“energy spectrum total value(ESTV)”.The rapid diagnostic method not only avoids tedious burnup calculations,but also provides a direction for optimization.The optimal irradiation schemes for producing ^(252)Cf,^(244)Cm,^(242)Cm,and ^(238)Pu are determined based on a rapid diagnostic method.Optimal irradiation schemes can significantly improve production efficiency.Compared with the initial scheme,the optimal scheme improved the production efficiency of ^(238)Pu by 7.41 times;^(242)Cm,11.98 times;^(244)Cm,65.20 times;and ^(252)Cf,15.08 times.Thus,a refined analysis of transplutonium isotope production is conducted and provides a theoretical basis for improving production efficiency.

Design of policy mechanism to promote cleaner production in China

In order to promote cleaner production in China, a package of policy options was presented based on the identification of the barriers encountered in the CP demonstration project and the effectiveness and feasibility of policy options for the CP implementation were analyzed. Furthermore, the policy mechanism framework was given, which composes of compulsory, economic incentive, social pressure and supportive function. Finally, the implementation strategy of the policy mechanism, in which the emphasis will be changed from compulsory policy options towards economic and social pressure policy options, was proposed.

Evolutionary-assisted reinforcement learning for reservoir real-time production optimization under uncertainty

Production optimization has gained increasing attention from the smart oilfield community because it can increase economic benefits and oil recovery substantially.While existing methods could produce high-optimality results,they cannot be applied to real-time optimization for large-scale reservoirs due to high computational demands.In addition,most methods generally assume that the reservoir model is deterministic and ignore the uncertainty of the subsurface environment,making the obtained scheme unreliable for practical deployment.In this work,an efficient and robust method,namely evolutionaryassisted reinforcement learning(EARL),is proposed to achieve real-time production optimization under uncertainty.Specifically,the production optimization problem is modeled as a Markov decision process in which a reinforcement learning agent interacts with the reservoir simulator to train a control policy that maximizes the specified goals.To deal with the problems of brittle convergence properties and lack of efficient exploration strategies of reinforcement learning approaches,a population-based evolutionary algorithm is introduced to assist the training of agents,which provides diverse exploration experiences and promotes stability and robustness due to its inherent redundancy.Compared with prior methods that only optimize a solution for a particular scenario,the proposed approach trains a policy that can adapt to uncertain environments and make real-time decisions to cope with unknown changes.The trained policy,represented by a deep convolutional neural network,can adaptively adjust the well controls based on different reservoir states.Simulation results on two reservoir models show that the proposed approach not only outperforms the RL and EA methods in terms of optimization efficiency but also has strong robustness and real-time decision capacity.

Optimal Design for Main Dimensions of a 75000t Floating Production and Storage Vessel

The basis, process and results of the demonstration of the main dimensions of a 75000t floating production and storage vessel are discussed in this paper. A simple but reliable orthogonal design method is applied in the main dimension optimization. The ideas of gradual approximation and feedback from various aspects are put into effect. During the demonstration, in order to make the model tally with the actual situation, the draft design is closely related to the computational analysis, so that the demonstration model can be verified at any time; the handling of the overall system is closely related to the research of each item, which is beneficial not only to the mastery of various regularities, but also to the balance of decisions. Finally, according to the computational results and the regularities obtained from analysis, the main dimensions are determined.

Empirical Analysis of China's Food Policy Impact on Grain Production

As to China with the characteristic of huge population and limited arable land,grain production has always been the most fundamental issue to national interests and people's livelihood,so China's government constantly enact food policy to promote and secure grain production. Based on the viewpoint of macro perspective,the paper firstly summarized the historical changes of food policy in China,secondly elaborated the status quo of the food policy,and thirdly made Granger Causality Test for food policy impact on grain production. Granger Causality Test result shows that fiscal expenditure for agriculture is granger cause for growth of grain production,so that it proves the positive role of the food policy on grain production. At last,the paper put forward suggestions for improving food policy in the future.

Hybrid data-driven framework for shale gas production performance analysis via game theory, machine learning, and optimization approaches

A comprehensive and precise analysis of shale gas production performance is crucial for evaluating resource potential,designing a field development plan,and making investment decisions.However,quantitative analysis can be challenging because production performance is dominated by the complex interaction among a series of geological and engineering factors.In fact,each factor can be viewed as a player who makes cooperative contributions to the production payoff within the constraints of physical laws and models.Inspired by the idea,we propose a hybrid data-driven analysis framework in this study,where the contributions of dominant factors are quantitatively evaluated,the productions are precisely forecasted,and the development optimization suggestions are comprehensively generated.More specifically,game theory and machine learning models are coupled to determine the dominating geological and engineering factors.The Shapley value with definite physical meaning is employed to quantitatively measure the effects of individual factors.A multi-model-fused stacked model is trained for production forecast,which provides the basis for derivative-free optimization algorithms to optimize the development plan.The complete workflow is validated with actual production data collected from the Fuling shale gas field,Sichuan Basin,China.The validation results show that the proposed procedure can draw rigorous conclusions with quantified evidence and thereby provide specific and reliable suggestions for development plan optimization.Comparing with traditional and experience-based approaches,the hybrid data-driven procedure is advanced in terms of both efficiency and accuracy.