Borel状态空间中平均零和随机博弈的新条件

本文研究Borel状态空间的离散时间Markov平均博弈.对报酬函数可以无界的一般情形,本文用平均最优双不等式取代相应的Shapley方程,提出比现有的几何遍历性条件更弱的新条件.在此新的条件下,本文建立上述平均最优双不等式的可解性,并由此证明平均博弈的值和Nash均衡策略的存在性.进而,在较强的几何遍历性条件下,用本文的最优双不等式,证明Shapley方程的可解性.最后,用电力系统与金融保险中的例子验证本文的条件,阐明本文的结果.

First passage Markov decision processes with constraints and varying discount factors

This paper focuses on the constrained optimality problem （COP） of first passage discrete-time Markov decision processes （DTMDPs） in denumerable state and compact Borel action spaces with multi-constraints, state-dependent discount factors, and possibly unbounded costs. By means of the properties of a so-called occupation measure of a policy, we show that the constrained optimality problem is equivalent to an （infinite-dimensional） linear programming on the set of occupation measures with some constraints, and thus prove the existence of an optimal policy under suitable conditions. Furthermore, using the equivalence between the constrained optimality problem and the linear programming, we obtain an exact form of an optimal policy for the case of finite states and actions. Finally, as an example, a controlled queueing system is given to illustrate our results.

Convergence of Markov decision processes with constraints and state-action dependent discount factors

This paper is concerned with the convergence of a sequence of discrete-time Markov decision processes(DTMDPs)with constraints,state-action dependent discount factors,and possibly unbounded costs.Using the convex analytic approach under mild conditions,we prove that the optimal values and optimal policies of the original DTMDPs converge to those of the"limit"one.Furthermore,we show that any countablestate DTMDP can be approximated by a sequence of finite-state DTMDPs,which are constructed using the truncation technique.Finally,we illustrate the approximation by solving a controlled queueing system numerically,and give the corresponding error bound of the approximation.

Optimal stopping time on discounted semi-Markov processes

This paper attempts to study the optimal stopping time for semi- Markov processes (SMPs) under the discount optimization criteria with unbounded cost rates. In our work, we introduce an explicit construction of the equivalent semi-Markov decision processes (SMDPs). The equivalence is embodied in the expected discounted cost functions of SMPs and SMDPs, that is, every stopping time of SMPs can induce a policy of SMDPs such that the value functions are equal, and vice versa. The existence of the optimal stopping time of SMPs is proved by this equivalence relation. Next, we give the optimality equation of the value function and develop an effective iterative algorithm for computing it. Moreover, we show that the optimal and ε-optimal stopping time can be characterized by the hitting time of the special sets. Finally, to illustrate the validity of our results, an example of a maintenance system is presented in the end.

TOTAL REWARD CRITERIA FOR UNCONSTRAINED/CONSTRAINED CONTINUOUS-TIME MARKOV DECISION PROCESSES

This paper studies denumerable continuous-time Markov decision processes with expected total reward criteria. The authors first study the unconstrained model with possible unbounded transition rates, and give suitable conditions on the controlled system＇s primitive data under which the authors show the existence of a solution to the total reward optimality equation and also the existence of an optimal stationary policy. Then, the authors impose a constraint on an expected total cost, and consider the associated constrained model. Basing on the results about the unconstrained model and using the Lagrange multipliers approach, the authors prove the existence of constrained-optimal policies under some additional conditions. Finally, the authors apply the results to controlled queueing systems.

Correction to a proposition related to ergodicity of Markov chains

Proposition 5.5.6(ii)in the book Markov Chains and Stochastic Stability(2nd ed,Cambridge Univ.Press,2009)has been used in the proof of a theorem about ergodicity of Markov chains.Unfortunately,an example in this paper shows that this proposition is not always true.Thus,a correction of this proposition is provided.