Analysis of a POMDP Model for an Optimal Maintenance Problem with Multiple Imperfect Repairs

I consider a system whose deterioration follows a discrete-time and discrete-state Markov chain with an absorbing state. When the system is put into practice, I may select operation (wait), imperfect repair, or replacement at each discrete-time point. The true state of the system is not known when it is operated. Instead, the system is monitored after operation and some incomplete information concerned with the deterioration is obtained for decision making. Since there are multiple imperfect repairs, I can select one option from them when the imperfect repair is preferable to operation and replacement. To express this situation, I propose a POMDP model and theoretically investigate the structure of an optimal maintenance policy minimizing a total expected discounted cost for an unbounded horizon. Then two stochastic orders are used for the analysis of our problem.

OPTIMAL HARVESTING POLICY FOR INSHORE-OFFSHORE FISHERY MODEL WITH IMPULSIVE DIFFUSION

This article studies the inshore-offshore fishery model with impulsive diffusion. The existence and global asymptotic stability of both the trivial periodic solution and the positive periodic solution are obtained. The complexity of this system is also analyzed. Moreover, the optimal harvesting policy are given for the inshore subpopulation, which includes the maximum sustainable yield and the corresponding harvesting effort.

THE OPTIMAL STRATEGY FOR INSURANCE COMPANY UNDER THE INFLUENCE OF TERMINAL VALUE

This paper considers a model of an insurance company which is allowed to invest a risky asset and to purchase proportional reinsurance. The objective is to find the policy which maximizes the expected total discounted dividend pay-out until the time of bankruptcy and the terminal value of the company under liquidity constraint. We find the solution of this problem via solving the problem with zero terminal value. We also analyze the influence of terminal value on the optimal policy.

Optimal Static Partition Configuration in ARINC653 System

ARINC653 systems, which have been widely used in avionics industry, are an important class of safety-critical applications. Partitions are the core concept in the Arinc653 system architecture. Due to the existence of partitions, the system designer must allocate adequate time slots statically to each partition in the design phase. Although some time slot allocation policies could be borrowed from task scheduling policies, no existing literatures give an optimal allocation policy. In this paper, we present a partition configuration policy and prove that this policy is optimal in the sense that if this policy fails to configure adequate time slots to each partition, nor do other policies. Then, by simulation, we show the effects of different partition configuration policies on time slot allocation of partitions and task response time, respectively.

RECURSIVE UTILITY,PRODUCTIVE GOVERNMENT EXPENDITURE AND OPTIMAL FISCAL POLICY

This paper employs a stochastic endogenous growth model extended to the case of a recursive utility function which can disentangle intertemporal substitution from risk aversion to analyze productive government expenditure and optimal fiscal policy, particularly stresses the importance of factor income. First, the explicit solutions of the central planner＇s stochastic optimization problem are derived, the growth maximizing and welfare-maximizing government expenditure policies are obtained and their standing in conflict or coincidence depends upon intertemporal substitution. Second, the explicit solutions of the representative individual＇s stochastic optimization problem which permits to tax on capital income and labor income separately are derived ,and it is found that the effect of risk on growth crucially depends on the degree of risk aversion,the intertemporal elasticity of substitution and the capital income share. Finally, a flexible optimal tax policy which can be internally adjusted to a certain extent is derived, and it is found that the distribution of factor income plays an important role in designing the optimal tax policy.

Optimal switching policy for performance enhancement of distributed parameter systems based on event-driven control

This paper aims to improve the performance of a class of distributed parameter systems for the optimal switching of actuators and controllers based on event-driven control. It is assumed that in the available multiple actuators, only one actuator can receive the control signal and be activated over an unfixed time interval, and the other actuators keep dormant. After incorporating a state observer into the event generator, the event-driven control loop and the minimum inter-event time are ultimately bounded. Based on the event-driven state feedback control, the time intervals of unfixed length can be obtained. The optimal switching policy is based on finite horizon linear quadratic optimal control at the beginning of each time subinterval. A simulation example demonstrate the effectiveness of the proposed policy.

Optimal policy for controlling two-server queueing systems with jockeying

This paper studies the optimal policy for joint control of admission, routing, service, and jockeying in a queueing system consisting of two exponential servers in parallel.Jobs arrive according to a Poisson process.Upon each arrival, an admission/routing decision is made, and the accepted job is routed to one of the two servers with each being associated with a queue.After each service completion, the servers have an option of serving a job from its own queue, serving a jockeying job from another queue, or staying idle.The system performance is inclusive of the revenues from accepted jobs, the costs of holding jobs in queues, the service costs and the job jockeying costs.To maximize the total expected discounted return, we formulate a Markov decision process(MDP) model for this system.The value iteration method is employed to characterize the optimal policy as a hedging point policy.Numerical studies verify the structure of the hedging point policy which is convenient for implementing control actions in practice.

Distributive Disturbance and Optimal Policy in Stochastic Control Model

To investigate the equilibrium relationships between the volatility of capital and income, taxation, and ance in a stochastic control model, the uniqueness of the solution to this model was proved by using the method of dynamic programming under the introduction of distributive disturbance and elastic labor supply. Furthermore, the effects of two types of shocks on labor-leisure choice, economic growth rate and welfare were numerically analyzed, and then the optimal tax policy was derived.

B-Spline-Based Curve Fitting to Cam Pitch Curve Using Reinforcement Learning

Directly applying the B-spline interpolation function to process plate cams in a computer numerical control(CNC)system may produce verbose tool-path codes and unsmooth trajectories.This paper is devoted to addressing the problem of B-splinefitting for cam pitch curves.Considering that the B-spline curve needs to meet the motion law of the follower to approximate the pitch curve,we use the radial error to quantify the effects of thefitting B-spline curve and the pitch curve.The problem thus boils down to solving a difficult global optimization problem tofind the numbers and positions of the control points or data points of the B-spline curve such that the cumulative radial error between thefitting curve and the original curve is minimized,and this problem is attempted in this paper with a double deep Q-network(DDQN)reinforcement learning(RL)algorithm with data points traceability.Specifically,the RL envir-onment,actions set and current states set are designed to facilitate the search of the data points,along with the design of the reward function and the initialization of the neural network.The experimental results show that when the angle division value of the actions set isfixed,the proposed algorithm can maximize the number of data points of the B-spline curve,and accurately place these data points to the right positions,with the minimum average of radial errors.Our work establishes the theoretical foundation for studying splinefitting using the RL method.

Task assignment in ground-to-air confrontation based on multiagent deep reinforcement learning

The scale of ground-to-air confrontation task assignments is large and needs to deal with many concurrent task assignments and random events.Aiming at the problems where existing task assignment methods are applied to ground-to-air confrontation,there is low efficiency in dealing with complex tasks,and there are interactive conflicts in multiagent systems.This study proposes a multiagent architecture based on a one-general agent with multiple narrow agents(OGMN)to reduce task assignment conflicts.Considering the slow speed of traditional dynamic task assignment algorithms,this paper proposes the proximal policy optimization for task assignment of general and narrow agents(PPOTAGNA)algorithm.The algorithm based on the idea of the optimal assignment strategy algorithm and combined with the training framework of deep reinforcement learning(DRL)adds a multihead attention mechanism and a stage reward mechanism to the bilateral band clipping PPO algorithm to solve the problem of low training efficiency.Finally,simulation experiments are carried out in the digital battlefield.The multiagent architecture based on OGMN combined with the PPO-TAGNA algorithm can obtain higher rewards faster and has a higher win ratio.By analyzing agent behavior,the efficiency,superiority and rationality of resource utilization of this method are verified.

基于多智能体深度强化学习的无人机路径规划

为解决多无人机(unmanned aerial vehicle, UAV)在复杂环境下的路径规划问题,提出一个多智能体深度强化学习UAV路径规划框架.该框架首先将路径规划问题建模为部分可观测马尔可夫过程,采用近端策略优化算法将其扩展至多智能体,通过设计UAV的状态观测空间、动作空间及奖赏函数等实现多UAV无障碍路径规划;其次,为适应UAV搭载的有限计算资源条件,进一步提出基于网络剪枝的多智能体近端策略优化(network pruning-based multi-agent proximal policy optimization, NP-MAPPO)算法,提高了训练效率.仿真结果验证了提出的多UAV路径规划框架在各参数配置下的有效性及NP-MAPPO算法在训练时间上的优越性.

Cooperative multi-target hunting by unmanned surface vehicles based on multi-agent reinforcement learning

To solve the problem of multi-target hunting by an unmanned surface vehicle(USV)fleet,a hunting algorithm based on multi-agent reinforcement learning is proposed.Firstly,the hunting environment and kinematic model without boundary constraints are built,and the criteria for successful target capture are given.Then,the cooperative hunting problem of a USV fleet is modeled as a decentralized partially observable Markov decision process(Dec-POMDP),and a distributed partially observable multitarget hunting Proximal Policy Optimization(DPOMH-PPO)algorithm applicable to USVs is proposed.In addition,an observation model,a reward function and the action space applicable to multi-target hunting tasks are designed.To deal with the dynamic change of observational feature dimension input by partially observable systems,a feature embedding block is proposed.By combining the two feature compression methods of column-wise max pooling(CMP)and column-wise average-pooling(CAP),observational feature encoding is established.Finally,the centralized training and decentralized execution framework is adopted to complete the training of hunting strategy.Each USV in the fleet shares the same policy and perform actions independently.Simulation experiments have verified the effectiveness of the DPOMH-PPO algorithm in the test scenarios with different numbers of USVs.Moreover,the advantages of the proposed model are comprehensively analyzed from the aspects of algorithm performance,migration effect in task scenarios and self-organization capability after being damaged,the potential deployment and application of DPOMH-PPO in the real environment is verified.

FALSE FAILURE RETURNS： OPTIMAL PRICING AND RETURN POLICIES IN A DUAL-CHANNEL SUPPLY CHAIN

This paper aim is to examine the optimal pricing and return policies for false failure returns in a dual-channel supply chain. Four prevailing return policies in which a manufacturer both operates an E-shop and sells its product through a brick-and-mortar retailer are analyzed, i.e. （I） the manufacturer handlings E-shop＇s returns, while the retailer addresses brick-and-mortar store＇s returns （NR）; （II） the retailer tackles the whole （both E-shop＇s and brick-and-mortar store＇s） returns （ORR）; （III） the manufacturer tackles the whole returns （ORM）; and （IV） the manufacturer and the retailer are jointly responsible for the whole returns （RRM）. Firstly, the optimal pricing and return policies comparing these four scenarios under uniform-pricing strategy are presented. Our conclusions show that the ORR is an optimal return policy. Compared with the NR, consumers will get a lower product pricing under the ORR and a higher product pricing under the ORM. With regard to the RRM, the product pricing is depended on consumer preference, return-rates of the E-shop and the brick-and-mortar store. Then, the optimal pricing and return policies are analyzed under differential-pricing strategy by conducting two-stage sequential games between the manufacturer and the retailer. The findings show that if consumers in the market prefer to purchase via the E-shop, the ORR is an optimal return policy. Otherwise, the NR is the optimal return policy. Compared with the NR, the ORR retailer＇s product pricing will rely on the retailer＇s and the manufacturer＇s return-costs; the RRM retailer＇s product pricing will depend on the return-costs of the retailer and the manufacturer, the return-rates of the E-shop and the brick-and-mortar store and so on. Finally, the influences of the manufacturer and the retailer establishing a Buy-back contract are discussed. Our results illustrated that the Buy-back contract doesn＇t affect optimal pricing and return policies under both the uniform and the differential pricing strategies.

Multi-agent reinforcement learning for edge information sharing in vehicular networks

To guarantee the heterogeneous delay requirements of the diverse vehicular services,it is necessary to design a full cooperative policy for both Vehicle to Infrastructure(V2I)and Vehicle to Vehicle(V2V)links.This paper investigates the reduction of the delay in edge information sharing for V2V links while satisfying the delay requirements of the V2I links.Specifically,a mean delay minimization problem and a maximum individual delay minimization problem are formulated to improve the global network performance and ensure the fairness of a single user,respectively.A multi-agent reinforcement learning framework is designed to solve these two problems,where a new reward function is proposed to evaluate the utilities of the two optimization objectives in a unified framework.Thereafter,a proximal policy optimization approach is proposed to enable each V2V user to learn its policy using the shared global network reward.The effectiveness of the proposed approach is finally validated by comparing the obtained results with those of the other baseline approaches through extensive simulation experiments.

Robust analysis of discounted Markov decision processes with uncertain transition probabilities

Optimal policies in Markov decision problems may be quite sensitive with regard to transition probabilities.In practice,some transition probabilities may be uncertain.The goals of the present study are to find the robust range for a certain optimal policy and to obtain value intervals of exact transition probabilities.Our research yields powerful contributions for Markov decision processes(MDPs)with uncertain transition probabilities.We first propose a method for estimating unknown transition probabilities based on maximum likelihood.Since the estimation may be far from accurate,and the highest expected total reward of the MDP may be sensitive to these transition probabilities,we analyze the robustness of an optimal policy and propose an approach for robust analysis.After giving the definition of a robust optimal policy with uncertain transition probabilities represented as sets of numbers,we formulate a model to obtain the optimal policy.Finally,we define the value intervals of the exact transition probabilities and construct models to determine the lower and upper bounds.Numerical examples are given to show the practicability of our methods.

STUDY ON THE OPTIMIZATION OF TRANSPORT CONTROL POLICY IN COMMUNICATION NETWORK

In communication networks with policy-based Transport Control on-Demand (TCoD) function,the transport control policies play a great impact on the network effectiveness. To evaluate and optimize the transport policies in communication network,a policy-based TCoD network model is given and a comprehensive evaluation index system of the network effectiveness is put forward from both network application and handling mechanism perspectives. A TCoD network prototype system based on Asynchronous Transfer Mode/Multi-Protocol Label Switching (ATM/MPLS) is introduced and some experiments are performed on it. The prototype system is evaluated and analyzed with the comprehensive evaluation index system. The results show that the index system can be used to judge whether the communication network can meet the application requirements or not,and can provide references for the optimization of the transport policies so as to improve the communication network effectiveness.

A generalized geometric process based repairable system model with bivariate policy

The maintenance model of simple repairable system is studied.We assume that there are two types of failure,namely type Ⅰ failure(repairable failure)and type Ⅱ failure(irrepairable failure).As long as the type Ⅰ failure occurs,the system will be repaired immediately,which is failure repair(FR).Between the(n-1)th and the nth FR,the system is supposed to be preventively repaired(PR)as the consecutive working time of the system reaches λ^(n-1) T,where λ and T are specified values.Further,we assume that the system will go on working when the repair is finished and will be replaced at the occurrence of the Nth type Ⅰ failure or the occurrence of the first type Ⅱ failure,whichever occurs first.In practice,the system will degrade with the increasing number of repairs.That is,the consecutive working time of the system forms a decreasing generalized geometric process(GGP)whereas the successive repair time forms an increasing GGP.A simple bivariate policy(T,N)repairable model is introduced based on GGP.The alternative searching method is used to minimize the cost rate function C(N,T),and the optimal(T,N)^(*) is obtained.Finally,numerical cases are applied to demonstrate the reasonability of this model.

Inventory Model with Supply Disruption Risk and Service Level Constraint

We develop a newsvender model with service level constraint for oneproduct and multi-supplier system, where the supplies are susceptible to disruption risks. We prove the convexity of the programming problem,and provide the necessary and sufficient conditions for the optimal order quantities. Through analysis or numerical experiments we observe the effects of the disrupt risk and the service level constraint on the optimal policy. We propose some managerial insights in managing the multi-source supply chain with disruption risks.

Scheduling and control of failure prone systems with demand uncertainty and job overlaps

Given that the overlapping of jobs is permitted, the paper studies the scheduling and control of failure prone production systems, i.e. so-called settings with demand uncertainty and job overlaps. Because a variable demand resource is involved in the production and corrective maintenance control problems of the system, which switched randomly between zero and a maximum level, it is difficult to obtain the analytical solutions of the optimal single hedging point policy. An asymptotic optimal scheduling policy is presented and a double hedging point policy is offered to control simultaneously the production rate and the corrective maintenance rate of the system. The corresponding analytical solutions and approximate solutions are obtained. Considering the relationship of production, corrective maintenance and demand variable, an approximate optimal single hedging point control policy is proposed. Numerical results are presented.

Optimal harvesting of a fuzzy water hyacinth-fish model with Kuznets curve effect

In this paper,we consider the effects of environmental perturbations such as the variations of temperature and climate on the growth of water hyacinth and fish species,as well as the economic and application values of these two species.We first present a triangular fuzzy water hyacinth-fish ecological model with Kuznets curve effect and linear harvesting.Then,we discuss the existence of the equilibria and their stability.It is then followed by analyzing the existence of bionomic equilibrium and determining the optimal harvesting policy via Pontryagin's Maximum Principle.Finally,numerical simulations are performed to validate the theoretical results.Our research indicates that fuzzy parameters may affect the optimal harvesting strategy.