Markov decision processes on Borel spaces with total cost and random horizon. (English) Zbl 1317.90316

The paper deals with Markov decision processes (MDPs) on Borel spaces with possibly unbounded costs. It was motivated by the study of the discounted optimal control problem given in the book by M. L. Puterman [Markov decision processes: discrete stochastic dynamic programming. New York, NY: John Wiley & Sons (1994; Zbl 0829.90134)]. In the book it is proved that the discounted control problem can be treated as a control problem where the horizon is a random variable, which is supposed to follow a geometric distribution independent of the process. The results of the paper are obtained with the help of a dynamic programming approach. They permit working with discounted control problem with varying-time discount factor, possibly depending on the state of the system and the corresponding action as well. To illustrate the theory developed, a version of the linear-quadratic model with a random horizon and a logarithm consumption-investment model are presented.


90C40 Markov and semi-Markov decision processes
49L20 Dynamic programming in optimal control and differential games
90C39 Dynamic programming
60J25 Continuous-time Markov processes on general state spaces


Zbl 0829.90134
Full Text: DOI


[1] Puterman, M.L.: Markov Decision Process: Discrete Stochastic Dynamic Programming. Wiley, New York (1994) · Zbl 0829.90134
[2] Baüerle, N., Rieder, U.: Markov Decision Processes with Applications to Finance. Springer, New York (2010)
[3] Kozlowski, E.: The linear-quadratic stochastic optimal control problem with random horizon at the finite number of infinitesimal events. Ann. UMCS Inform. 1, 103-115 (2010) · Zbl 1284.93263
[4] Levhari, D., Mirman, L.J.: Savings and consumption with uncertain horizon. J. Polit. Econ. 85, 265-281 (1977)
[5] Bather, J.: Decision Theory: An Introduction to Dynamic Programming and Sequential Decision. Wiley, New York (2000) · Zbl 0955.91009
[6] Chatterjee, D., Cinquemani, E., Chaloulos, G., Lygeros, J.: Stochastic control up to a hitting time: optimality and Rolling-Horizon implementation (2009). arXiv:0806.3008
[7] Iida, T., Mori, M.: Markov decision processes with random horizon. J. Oper. Res. Soc. Jpn. 39, 592-603 (1996) · Zbl 0874.90189
[8] Hernández-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Springer, New York (1996)
[9] Guo, X., Hernandez-del-Valle, A., Hernández-Lerma, O.: Nonstationary discrete-time deterministic and stochastic control systems with infinite horizon. Int. J. Control 83, 1751-1757 (2010) · Zbl 1213.49035
[10] Hinderer, K.: Foundation of Non-Stationary Dynamic Programming with Discrete Time Parameter. Springer, New York (1970) · Zbl 0202.18401
[11] Bertsekas, D.P., Shreve, S.E.: Stochastic Optimal Control: the Discrete Time Case. Academic Press, Massachusetts (1978) · Zbl 0471.93002
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.