zbMATH — the first resource for mathematics

A characterization of the optimal risk-sensitive average cost in finite controlled Markov chains. (English) Zbl 1076.93045
The paper deals with optimal control problems for Markov chains with finite state and action spaces when the performance index is given by the long-run risk-sensitive average cost criterion. It is assumed that the transition law of the controlled chain satisfies the simultaneous Doeblin condition.
The main result characterizes for any positive risk-sensitivity coefficient the optimal value function as the infimum of a family of functions on the state space. It is then shown that for large values of the risk-sensitivity coefficient the optimal cost is not necessarily constant and the optimality equation may have no solution at all.

93E20 Optimal stochastic control
60F10 Large deviations
93C55 Discrete-time control/observation systems
60J05 Discrete-time Markov processes on general state spaces
Full Text: DOI arXiv
[1] Bielecki, T., Hernández-Hernández, D. and Pliska, S. (1999). Risk sensitive control of finite state Markov chains in discrete time, with applications to portfolio management. Math. Methods Oper. Res. 50 167–188. · Zbl 0959.91029 · doi:10.1007/s001860050094
[2] Bielecki, T. and Pliska, S. (1999). Risk sensitive dynamic asset management. Appl. Math. Optim. 39 337–360. · Zbl 0984.91047 · doi:10.1007/s002459900110
[3] Cavazos-Cadena, R. and Fernández-Gaucherand, E. (1999). Controlled Markov chains with risk-sensitive criteria: Average cost, optimality equations, and optimal solutions. Math. Methods Oper. Res. 49 299–324. · Zbl 0953.93077
[4] Cavazos-Cadena, R. and Fernández-Gaucherand, E. (2001). Risk-sensitive optimal control in communicating average Markov decision chains. In Modeling Uncertainty : An Examination of Stochastic Theory , Methods , and Applications (M. Dror, P. L’Ecuyer and F. Szydarovszky, eds.) 515–553. Kluwer Academic, Dordrecht. · doi:10.1007/0-306-48102-2_22
[5] Cavazos-Cadena, R. and Hernández-Hernández, D. (2002). Solution to the risk-sensitive average optimality equation in communicating Markov decision chains with finite state space: An alternative approach. Math. Methods Oper. Res. 56 473–479. · Zbl 1040.93070 · doi:10.1007/s001860200229
[6] Cavazos-Cadena, R. and Hernández-Hernández, D. (2004). A characterization of exponential functionals in finite Markov chains. Math. Methods Oper. Res. · Zbl 1072.60053 · doi:10.1007/s001860400373
[7] Dembo, A. and Zeitouni, O. (1993). Large Deviations Techniques and Applications . Jones and Bartlett, Boston. · Zbl 0793.60030
[8] Di Masi, G. B. and Stettner, L. (1999). Risk-sensitive control of discrete-time Markov processes with infinite horizon. SIAM J. Control Optim. 38 61–78. · Zbl 0946.93043 · doi:10.1137/S0363012997320614
[9] Donsker, M. D. and Varadhan, S. R. (1975). Asymptotic evaluation of certain Markov process expectations for large time, I, II, III. Comm. Pure Appl. Math. 28 1–45, 279–301; 29 389–461. · Zbl 0323.60069
[10] Dynkin, E. B. and Yushkevich, A. A. (1979). Controlled Markov Processes . Springer, Berlin. · Zbl 0073.34801
[11] Fleming, W. H. and Hernández-Hernández, D. (1997). Risk sensitive control of finite state machines on an infinite horizon I. SIAM J. Control Optim. 35 1790–1810. · Zbl 0891.93085 · doi:10.1137/S0363012995291622
[12] Fleming, W. H. and Sheu, S. (1999). Optimal long term growth rate of expected utility of wealth. Ann. Appl. Probab. 9 871–903. · Zbl 0962.91036 · doi:10.1214/aoap/1029962817
[13] Gale, D. (1968). A mathematical theory of optimal economic development. Bull. Amer. Math. Soc. 74 207–223. · Zbl 0172.44307 · doi:10.1090/S0002-9904-1968-11891-9
[14] Hernández-Hernández, D. and Marcus, S. I. (1996). Risk sensitive control of Markov processes in countable state space. Systems Control Lett. 29 147–155. [Corrigendum (1998) 34 105–106.] · Zbl 0866.93101 · doi:10.1016/S0167-6911(96)00051-5
[15] Hernández-Lerma, O. (1988). Adaptive Markov Control Processes . Springer, New York. · Zbl 0646.90090
[16] Hordjik, A. (1974). Dynamic Programming and Potential Theory . Math. Centrum, Amsterdam.
[17] Howard, A. R. and Matheson, J. E. (1972). Risk-sensitive Markov decision processes. Managment Sciences 18 356–369. · Zbl 0238.90007 · doi:10.1287/mnsc.18.7.356
[18] Puterman M. L. (1994). Markov Decision Processes : Discrete Stochastic Dynamic Programming . Wiley, New York. · Zbl 0829.90134
[19] Thomas L. C. (1980). Connectedness conditions for denumerable state Markov decision processes. In Recent Advances in Markov Decision Processes (R. Hartles, L. C. Thomas and D. J. White, eds.) 181–204. Academic Press, New York.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.