Variance minimization for continuous-time Markov decision processes: two approaches

This paper studies the limit average variance criterion for continuous-time Markov decision processes in Polish spaces. Based on two approaches, this paper proves not only the existence of solutions to the variance minimization optimality equation and the existence of a variance minimal policy that is canonical, but also the existence of solutions to the two variance minimization optimality inequalities and the existence of a variance minimal policy which may not be canonical. An example is given to illustrate all of our conditions.

Development of Optimal Maintenance Policies for Offshore Wind Turbine Gearboxes Based on the Non-homogeneous Continuous-Time Markov Process

Gearbox in offshore wind turbines is a component with the highest failure rates during operation. Analysis of gearbox repair policy that includes economic considerations is important for the effective operation of offshore wind farms. From their initial perfect working states, gearboxes degrade with time, which leads to decreased working efficiency. Thus, offshore wind turbine gearboxes can be considered to be multi-state systems with the various levels of productivity for different working states. To efficiently compute the time-dependent distribution of this multi-state system and analyze its reliability, application of the nonhomogeneous continuous-time Markov process(NHCTMP) is appropriate for this type of object. To determine the relationship between operation time and maintenance cost, many factors must be taken into account, including maintenance processes and vessel requirements. Finally, an optimal repair policy can be formulated based on this relationship.

Variance Optimization for Continuous-Time Markov Decision Processes

This paper considers the variance optimization problem of average reward in continuous-time Markov decision process (MDP). It is assumed that the state space is countable and the action space is Borel measurable space. The main purpose of this paper is to find the policy with the minimal variance in the deterministic stationary policy space. Unlike the traditional Markov decision process, the cost function in the variance criterion will be affected by future actions. To this end, we convert the variance minimization problem into a standard (MDP) by introducing a concept called pseudo-variance. Further, by giving the policy iterative algorithm of pseudo-variance optimization problem, the optimal policy of the original variance optimization problem is derived, and a sufficient condition for the variance optimal policy is given. Finally, we use an example to illustrate the conclusion of this paper.

Stability Estimation for Markov Control Processes with Discounted Cost

This article explores controllable Borel spaces, stationary, homogeneous Markov processes, discrete time with infinite horizon, with bounded cost functions and using the expected total discounted cost criterion. The problem of the estimation of stability for this type of process is set. The central objective is to obtain a bounded stability index expressed in terms of the Lévy-Prokhorov metric;likewise, sufficient conditions are provided for the existence of such inequalities.

Asymptotic Evaluations of the Stability Index for a Markov Control Process with the Expected Total Discounted Reward Criterion

In this work, for a control consumption-investment process with the discounted reward optimization criteria, a numerical estimate of the stability index is made. Using explicit formulas for the optimal stationary policies and for the value functions, the stability index is explicitly calculated and through statistical techniques its asymptotic behavior is investigated (using numerical experiments) when the discount coefficient approaches 1. The results obtained define the conditions under which an approximate optimal stationary policy can be used to control the original process.

连续时间Markov控制过程的平均代价最优鲁棒控制策略

在Markov性能势基础上 ,研究了一类转移速率不确定但受紧集约束的遍历连续时间Markov控制过程 (CTMCP)的鲁棒控制问题 .根据系统的遍历性 ,平均代价Poisson方程的解可被看作是性能势的一种定义 .在平均代价准则下 ,优化控制的目标是选择一个平稳策略使得系统在参数最坏取值下能获得最小无穷水平平均代价 ,据此论文给出了求解最优鲁棒控制策略的策略迭代 (PI)算法 ,并详细讨论了算法的收敛性 .

随机模型检测连续时间Markov过程

功能正确和性能可满足是复杂系统可信要求非常重要的两个方面。从定性验证和定量分析相结合的角度,对复杂并发系统进行功能验证和性能分析,统一地评估系统是否可信。连续时间Markov决策过程CTMDP(Continu-ous-time Markov decision process)能够统一刻画复杂系统的概率选择、随机时间及不确定性等重要特征。提出用CT-MDP作为系统定性验证和定量分析模型,将复杂系统的功能验证和性能分析转化为CTMDP中的可达概率求解,并证明验证过程的正确性,最终借助模型检测器MRMC(Markov Reward Model Checker)实现模型检测。理论分析表明,提出的针对CTMDP模型的验证需求是必要的,验证思路和方法具有可行性。

基于离散Markov决策过程的发电公司多阶段决策

采用离散时间Markov决策过程(DTMDP)对以多阶段总利润最优为目标的发电公司决策问题进行研究.市场环境下,发电公司根据自身条件,其竞争策略可以是价格的接受者,也可以是价格的制定者.考虑了发电公司不同策略情况下市场均衡状态间的转换概率,分别给出了发电公司作为价格接受者和价格制定者时的多阶段决策模型.通过算例验证了所提模型的有效性和可行性.

基于Markov模型的离散事件系统稳态与暂态的分析

利用马尔科夫链的结果,在离散事件系统(DES)逻辑层次的自动机模型基础上,对DES的Markov模型的稳态和暂态特性,分别从时间参数连续和离散的情况下,分四个情况进行了分析,通过实例对系统遍历性提出了一条更简单的且在连续和离散时间参数情况下都通用的判定规则,并利用Kolmogorov向后或向前方程,对连续时间参数DES的暂态特性进行了分析和计算。关于时间参数连续DES的稳态分布着重给出了生灭过程模型稳态分布的计算方法。讨论了DES模型统计性能层次与逻辑层次之间的联系。

Markov控制过程基于性能势仿真的并行优化

Markov控制过程是研究随机离散事件动态系统性能优化问题的一个重要模型,并在许多实际工程问题中有着广泛的应用。在Markov性能势理论的基础上,我们讨论了一类连续时间Markov控制过程在紧致行动集上的性能优化仿真问题。由于实际系统的状态空间往往非常巨大,通常的串行仿真算法,可能耗时过长,也可能由于硬件限制而无法实现,故我们提出了一种基于性能势的并行仿真优化算法,来寻找系统的最优平稳策略。一个仿真实例表明该算法有较好的运行效率。该算法可应用于大规模实际系统的性能优化。

First passage times for multidimensional denumerable state Markov processes

For a general multidimensional denumerable state Markov process with any initial state probability vector, the probability density function and its LS transform of the first passage time to a certain given state set are obtained and the algorithms for them are derived. It is proved that the resulting errors of the algorithms are both uniform in their respective arguments.Some numerical results are presented.

Almost sure, L1-and L2-growth behavior of supercritical multi-type continuous state and continuous time branching processes with immigration

Under a first order moment condition on the immigration mechanism,we show that an appropriately scaled supercritical and irreducible multi-type continuous state and continuous time branching process with immigration(CBI process)converges almost surely.If an x log(x)moment condition on the branching mechanism does not hold,then the limit is zero.If this x log(x)moment condition holds,then we prove L1 convergence as well.The projection of the limit on any left non-Perron eigenvector of the branching mean matrix is vanishing.If,in addition,a suitable extra power moment condition on the branching mechanism holds,then we provide the correct scaling for the projection of a CBI process on certain left non-Perron eigenvectors of the branching mean matrix in order to have almost sure and L1 limit.Moreover,under a second order moment condition on the branching and immigration mechanisms,we prove L2 convergence of an appropriately scaled process and the above-mentioned projections as well.A representation of the limits is also provided under the same moment conditions.

Maxima and sum for discrete and continuous time Gaussian processes

We study the asymptotic relation among the maximum of continuous weakly and strongly dependent stationary Gaussian process, the maximum of this process sampled at discrete time points, and the partial sum of this process. It is shown that these two extreme values and the sum are asymptotically independent if the grid of the discrete time points is sufficiently sparse and the Gaussian process is weakly dependent, and asymptotically dependent if the grid points are Pickands grids or dense grids.

A Stochastic SIVS Epidemic Model Based on Birth and Death Process

A new stochastic epidemic model, that is, a general continuous time birth and death chain model, is formulated based on a deterministic model including vaccination. We use continuous time Markov chain to construct the birth and death process. Through the Kolmogorov forward equation and the theory of moment generating function, the corresponding population expectations are studied. The theoretical result of the stochastic model and deterministic version is also given. Finally, numerical simulations are carried out to substantiate the theoretical results of random walk.

Average Sample-path Optimality for Continuous-time Markov Decision Processes in Polish Spaces

In this paper we study the average sample-path cost （ASPC） problem for continuous-time Markov decision processes in Polish spaces. To the best of our knowledge, this paper is a first attempt to study the ASPC criterion on continuous-time MDPs with Polish state and action spaces. The corresponding transition rates are allowed to be unbounded, and the cost rates may have neither upper nor lower bounds. Under some mild hypotheses, we prove the existence of （ε〉 0）-ASPC optimal stationary policies based on two different approaches： one is the ＂optimality equation＂ approach and the other is the ＂two optimality inequalities＂ approach.

CONVERGENCE OF CONTROLLED MODELS FOR CONTINUOUS-TIME MARKOV DECISION PROCESSES WITH CONSTRAINED AVERAGE CRITERIA

This paper attempts to study the convergence of optimal values and optimal policies of continuous-time Markov decision processes(CTMDP for short)under the constrained average criteria. For a given original model M_∞of CTMDP with denumerable states and a sequence {M_n} of CTMDP with finite states, we give a new convergence condition to ensure that the optimal values and optimal policies of {M_n} converge to the optimal value and optimal policy of M_∞as the state space Snof Mnconverges to the state space S_∞of M_∞, respectively. The transition rates and cost/reward functions of M_∞are allowed to be unbounded. Our approach can be viewed as a combination method of linear program and Lagrange multipliers.

基于距离信息的追逃策略:信念状态连续随机博弈

追逃问题的研究在对抗、追踪以及搜查等领域极具现实意义.借助连续随机博弈与马尔科夫决策过程(Markov decision process, MDP),研究使用测量距离求解多对一追逃问题的最优策略.在此追逃问题中,追捕群体仅领导者可测量与逃逸者间的相对距离,而逃逸者具有全局视野.追逃策略求解被分为追博弈与马尔科夫决策两个过程.在求解追捕策略时,通过分割环境引入信念区域状态以估计逃逸者位置,同时使用测量距离对信念区域状态进行修正,构建起基于信念区域状态的连续随机追博弈,并借助不动点定理证明了博弈平稳纳什均衡策略的存在性.在求解逃逸策略时,逃逸者根据全局信息建立混合状态下的马尔科夫决策过程及相应的最优贝尔曼方程.同时给出了基于强化学习的平稳追逃策略求解算法,并通过案例验证了该算法的有效性.

基于硬件同步的四态离散调制连续变量量子密钥分发

在连续变量量子密钥分发系统中,同步技术是确保通信双方时钟和数据一致的关键技术.本文通过巧妙设计发送端和接收端仪器的硬件时序,采用时域差拍探测方式和峰值采集技术,实验实现了可硬件同步的四态离散调制连续变量量子密钥分发.通信双方在设计好的硬件同步时序下可实现时钟的恢复和数据的自动对齐,无需借助软件算法实现数据的对齐.本文采用了加拿大滑铁卢大学Norbert Lütkenhaus研究组提出的针对连续变量离散调制协议的安全密钥速率计算方法.该方法需计算出接收端所测各种平移热态的一阶矩和二阶(非中心)矩,以此为约束条件结合凸优化算法可计算出安全密钥速率.计算过程中无需假设信道为线性信道,无需额外噪声的估算.密钥分发系统重复频率为10 MHz,传输距离为25 km,平均安全密钥比特率为24 kbit/s.本文提出的硬件同步方法无需过采样和软件帧同步,减小了系统的复杂度和计算量,在一定程度上降低了系统所需的成本、功耗和体积,有效地增强了连续变量量子密钥分发的实用性.

耦合级联失效系统可靠性建模与分析

现实生活中绝大多数系统并不是孤立存在的,如通信网和电网,它们相互依存、相互影响,这种系统间的耦合关系使得级联失效范围变得更广,导致级联过程更为复杂,从而影响整个系统可靠性及其正常运行。针对此问题,论文以电力通信系统为研究背景,给出了耦合系统转移率的解析表达,分析了元件负载增加影响元件故障率的级联失效效应和子系统间的相依关系,建立了耦合级联失效系统的可靠性模型,并证明了系统可靠度的计算方法和解析式结果。并且利用一个算例展示了耦合系统发生级联失效的具体过程,以验证该方法的有效性与可行性。本文为基于负载和时间的耦合系统的级联研究提供了新的思路,可拓展至不同的耦合关系、耦合强度以及不同的负载分配模式来进一步研究系统的级联失效过程以及可靠度分析。

Convergence of Markov decision processes with constraints and state-action dependent discount factors

This paper is concerned with the convergence of a sequence of discrete-time Markov decision processes(DTMDPs)with constraints,state-action dependent discount factors,and possibly unbounded costs.Using the convex analytic approach under mild conditions,we prove that the optimal values and optimal policies of the original DTMDPs converge to those of the"limit"one.Furthermore,we show that any countablestate DTMDP can be approximated by a sequence of finite-state DTMDPs,which are constructed using the truncation technique.Finally,we illustrate the approximation by solving a controlled queueing system numerically,and give the corresponding error bound of the approximation.