Markov control processes with randomized discounted cost. (English) Zbl 1126.90075

Summary: In this paper we consider Markov decision processes with discounted cost and a random rate in Borel spaces. We establish the dynamic programming algorithm in finite and infinite horizon cases. We provide conditions for the existence of measurable selectors. And we show an example of consumption-investment problem.


90C40 Markov and semi-Markov decision processes
93E20 Optimal stochastic control
Full Text: DOI


[1] Abbad M, Daoui C (2003) Hierarchical algorithms for discounted Markov decision processes. Math Methods Oper Res 58:237–245 · Zbl 1069.90106
[2] Altman E (1999) Constrained Markov decision processes. Chapman & Hall/CRC, Boca Raton · Zbl 0963.90068
[3] Bertsekas DP (1995) Dynamic programming and optimal control. Athena Scientific, Belmont · Zbl 0904.90170
[4] Berument H, Kilinc Z, Ozlale U (2004) The effects of different inflation risk prepius on interest rate spreads. Physica A 333:317–324
[5] Borkar VS (1991) Topics in controlled Markov chains. Longman, New York · Zbl 0725.93082
[6] Cai J, Dickson D, David CM (2004) Ruin probabilities with a Markov chain interest model. Insur Math Econ 35:513–525 · Zbl 1122.91340
[7] Feinberg EA, Shwartz A (1994) Markov decision models with weighted discounted criteria. Math Oper Res 19:152–168 · Zbl 0803.90123
[8] Feinberg EA, Shwartz A (1995) Constrained Markov decision models with weighted discounted rewards. Math Oper Res 20:302–320 · Zbl 0837.90120
[9] Feinberg EA, Shwartz A (1999) Constrained dynamic programming with two discount factors: applications and an algorithm. IEEE Trans Autom Control 42:628–631 · Zbl 0957.90127
[10] Gil A, Luis A (2004) Modelling the U.S. interest rate in terms of I(d) statistical models. Q Rev Econ Finance 44:475–486
[11] Haberman S, Sung J (2005) Optimal pension funding dynamics over infinite control horizon when stochastic rates of return are stationary. Insur Math Econ 36:103–116 · Zbl 1111.91023
[12] Hernández-Lerma O, González-Hernández J (2000) Constrained Markov control processes in Borel spaces: the discounted case. Math Meth Oper Res, 52:271–285 · Zbl 1032.90061
[13] Hernández-Lerma O, Lasserre JB (1996) Discrete-time Markov control processes: basic optimality criteria. Springer, Berlin Heidelberg New York
[14] Hinderer K (1970) Foundations of non-stationary dynamical programming with discrete time parameter Lecture notes in operation research, vol 33. Springer, Berlin Heidelberg New York · Zbl 0202.18401
[15] Hu Q (2003) Analysis for some properties of discrete time Markov decision processes. Optimization 52:495–505 · Zbl 1098.90082
[16] Kurano M, Song J, Hosaka M, Huang Y (1998) Controlled Markov set-chains with discounting. J Appl Prob 35:293–302 · Zbl 0911.90344
[17] Lee P, Rosenfield DB (2005) When to refinance a mortgage: a dynamic programming approach. Eur J Oper Res 166:266–277 · Zbl 1066.91022
[18] Lettau M, Uhlig H (1999) Rules of thumb versus dynamic programming. Am Econ Rev 89(1):148–174
[19] Liu K (1999) Weighted discounted Markov decision processes with perturbation. Acta Math Appl Sinica 15:183–189 · Zbl 0931.93075
[20] López-Martínez RR, Hernández-Lerma O (2003) The Lagrange approach to constrained Markov processes: a survey and extension of results. Morfismos 7:1–26
[21] Newell RG, Pizer WA (2003) Discounting the distant future: how much do uncertain rates increase valuation?. J Environ Econ Manage 46:52–71 · Zbl 1041.91502
[22] Mao X, Piunovskiy AB (2000) Strategic measure in optimal control problems for stochastic sequences. Stoch Anal Appl 18:755–776 · Zbl 0973.93060
[23] Michael K Ng (1999) A note on policy algorithms for discounted Markov decision problems. Oper Res Lett 25:195–197 · Zbl 0937.90117
[24] Ogaki M, Santaella JA (2000) The exchange rate and the term structure of interest rates in México. J Dev Econ 63:135–155
[25] Piunovskiy AB (1997) Optimal control of random sequences in problems with constraints. Kluwer, Boston · Zbl 0894.93001
[26] Puterman ML (1994) Markov decision processes: discrete stochastic dynamic programming. Wiley, New York · Zbl 0829.90134
[27] Sack B, Wieland V (2000) Interest-rate smoothing and optimal monetary policy: a review of recent empirical evidence. J Econ Bus 52:205–228
[28] Shwartz A (2001) Death and discounting. IEEE Trans Autom Control 46:644–647 · Zbl 1017.90122
[29] Stockey NL, Lucas RE Jr (1989) Recursive methods in economic dynamics. Harvard University Press, Cambridge
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.