Variance minimization for continuous-time Markov decision processes: two approaches

This paper studies the limit average variance criterion for continuous-time Markov decision processes in Polish spaces. Based on two approaches, this paper proves not only the existence of solutions to the variance minimization optimality equation and the existence of a variance minimal policy that is canonical, but also the existence of solutions to the two variance minimization optimality inequalities and the existence of a variance minimal policy which may not be canonical. An example is given to illustrate all of our conditions.

Variance Optimization for Continuous-Time Markov Decision Processes

This paper considers the variance optimization problem of average reward in continuous-time Markov decision process (MDP). It is assumed that the state space is countable and the action space is Borel measurable space. The main purpose of this paper is to find the policy with the minimal variance in the deterministic stationary policy space. Unlike the traditional Markov decision process, the cost function in the variance criterion will be affected by future actions. To this end, we convert the variance minimization problem into a standard (MDP) by introducing a concept called pseudo-variance. Further, by giving the policy iterative algorithm of pseudo-variance optimization problem, the optimal policy of the original variance optimization problem is derived, and a sufficient condition for the variance optimal policy is given. Finally, we use an example to illustrate the conclusion of this paper.

Non-Homogeneous Poisson Processes Applied to Count Data:A Bayesian Approach Considering Different Prior Distributions

This article discusses the Bayesian approach for count data using non-homogeneous Poisson processes, considering different prior distributions for the model parameters. A Bayesian approach using Markov Chain Monte Carlo (MCMC) simulation methods for this model was first introduced by [1], taking into account software reliability data and considering non-informative prior distributions for the parameters of the model. With the non-informative prior distributions presented by these authors, computational difficulties may occur when using MCMC methods. This article considers different prior distributions for the parameters of the proposed model, and studies the effect of such prior distributions on the convergence and accuracy of the results. In order to illustrate the proposed methodology, two examples are considered: the first one has simulated data, and the second has a set of data for pollution issues at a region in Mexico City.

A Markov regenerative process with recurrence time and its application

This study proposes a non-homogeneous continuous-time Markov regenerative process with recurrence times,in particular,forward and backward recurrence processes.We obtain the transient solution of the process in the form of a generalized Markov renewal equation.A distinguishing feature is that Markov and semi-Markov processes result as special cases of the proposed model.To model the credit rating dynamics to demonstrate its applicability,we apply the proposed stochastic process to Standard and Poor’s rating agency’s data.Further,statistical tests confirm that the proposed model captures the rating dynamics better than the existing models,and the inclusion of recurrence times significantly impacts the transition probabilities.

Applications of Dynamic-Equilibrium Continuous Markov Stochastic Processes to Elements of Survival Analysis

In this article, we summarize some results on invariant non-homogeneous and dynamic-equilibrium (DE) continuous Markov stochastic processes. Moreover, we discuss a few examples and consider a new application of DE processes to elements of survival analysis. These elements concern the stochastic quadratic-hazard-rate model, for which our work 1) generalizes the reading of its It? stochastic ordinary differential equation (ISODE) for the hazard-rate-driving independent (HRDI) variables, 2) specifies key properties of the hazard-rate function, and in particular, reveals that the baseline value of the HRDI variables is the expectation of the DE solution of the ISODE, 3) suggests practical settings for obtaining multi-dimensional probability densities necessary for consistent and systematic reconstruction of missing data by Gibbs sampling and 4) further develops the corresponding line of modeling. The resulting advantages are emphasized in connection with the framework of clinical trials of chronic obstructive pulmonary disease (COPD) where we propose the use of an endpoint reflecting the narrowing of airways. This endpoint is based on a fairly compact geometric model that quantifies the course of the obstruction, shows how it is associated with the hazard rate, and clarifies why it is life-threatening. The work also suggests a few directions for future research.

Average Sample-path Optimality for Continuous-time Markov Decision Processes in Polish Spaces

In this paper we study the average sample-path cost （ASPC） problem for continuous-time Markov decision processes in Polish spaces. To the best of our knowledge, this paper is a first attempt to study the ASPC criterion on continuous-time MDPs with Polish state and action spaces. The corresponding transition rates are allowed to be unbounded, and the cost rates may have neither upper nor lower bounds. Under some mild hypotheses, we prove the existence of （ε〉 0）-ASPC optimal stationary policies based on two different approaches： one is the ＂optimality equation＂ approach and the other is the ＂two optimality inequalities＂ approach.

CONVERGENCE OF CONTROLLED MODELS FOR CONTINUOUS-TIME MARKOV DECISION PROCESSES WITH CONSTRAINED AVERAGE CRITERIA

This paper attempts to study the convergence of optimal values and optimal policies of continuous-time Markov decision processes(CTMDP for short)under the constrained average criteria. For a given original model M_∞of CTMDP with denumerable states and a sequence {M_n} of CTMDP with finite states, we give a new convergence condition to ensure that the optimal values and optimal policies of {M_n} converge to the optimal value and optimal policy of M_∞as the state space Snof Mnconverges to the state space S_∞of M_∞, respectively. The transition rates and cost/reward functions of M_∞are allowed to be unbounded. Our approach can be viewed as a combination method of linear program and Lagrange multipliers.

Integral-type functionals of first hitting times for continuous-time Markov chains

We investigate integral-type functionals of the first hitting times for continuous-time Markov chains. Recursive formulas and drift conditions for calculating or bounding integral-type functionals are obtained. The connection between the subexponential integral-type functionals and the subexponential ergodicity is established. Moreover, these results are applied to the birth-death processes. Polynomial integral-type functionals and polynomial ergodicity are studied, and a sufficient criterion for a central limit theorem is also presented.

基于非均匀连续时间马尔可夫原理的海上浮式风机齿轮箱最优维修策略研究（英文）

Gearbox in offshore wind turbines is a component with the highest failure rates during operation. Analysis of gearbox repair policy that includes economic considerations is important for the effective operation of offshore wind farms. From their initial perfect working states, gearboxes degrade with time, which leads to decreased working efficiency. Thus, offshore wind turbine gearboxes can be considered to be multi-state systems with the various levels of productivity for different working states. To efficiently compute the time-dependent distribution of this multi-state system and analyze its reliability, application of the nonhomogeneous continuous-time Markov process(NHCTMP) is appropriate for this type of object. To determine the relationship between operation time and maintenance cost, many factors must be taken into account, including maintenance processes and vessel requirements. Finally, an optimal repair policy can be formulated based on this relationship.

On the Stability of Stochastic Jump Kinetics

Motivated by the lack of a suitable constructive framework for analyzing popular stochastic models of Systems Biology, we devise conditions for existence and uniqueness of solutions to certain jump stochastic differential equations (SDEs). Working from simple examples we find reasonable and explicit assumptions on the driving coefficients for the SDE representation to make sense. By “reasonable” we mean that stronger assumptions generally do not hold for systems of practical interest. In particular, we argue against the traditional use of global Lipschitz conditions and certain common growth restrictions. By “explicit”, finally, we like to highlight the fact that the various constants occurring among our assumptions all can be determined once the model is fixed. We show how basic long time estimates and some limit results for perturbations can be derived in this setting such that these can be contrasted with the corresponding estimates from deterministic dynamics. The main complication is that the natural path-wise representation is generated by a counting measure with an intensity that depends nonlinearly on the state.

Strong Approximations of Martingale Vectors and Their Applications in Markov-Chain Adaptive Designs

The strong approximations of a class of R^d-valued martingales are considered.The conditions usedin this paper are easier to check than those used in [3] and [9].As an application,the strong approximation ofa class of non-homogenous Markov chains is established,and the asymptotic properties are established for themulti-treatment Markov chain adaptive designs in clinical trials.

A primer on stochastic epidemic models:Formulation,numerical simulation,and analysis

Some mathematical methods for formulation and numerical simulation of stochastic epidemic models are presented.Specifically,models are formulated for continuous-time Markov chains and stochastic differential equations.Some well-known examples are used for illustration such as an SIR epidemic model and a host-vector malaria model.Analytical methods for approximating the probability of a disease outbreak are also discussed.