Average cost Markov control processes: Stability with respect to the Kantorovich metric. (English) Zbl 1176.60062

Gordienko, Evgueni; Lemus-Rodríguez, Enrique; Montes-de-Oca, Raúl

Math. Methods Oper. Res. 70, No. 1, 13-33 (2009).

Perturbations of a discrete-time Markov control process are studied on a general state space. The amount of perturbation is measured by means of the Kantorovich distance. It is assumed that an average (per unit of time on the infinite horizon) optimal control policy can be found for the perturbed (supposedly known) process, and that it is used to control the original (unperturbed) process. The one-stage cost is not assumed to be bounded. Under Lyapunov-like conditions, the authors find upper bounds for the average cost excess when such an approximation is used in place of the optimal (unknown) control policy. As an application of the inequalities found, the approximation by relevant empirical distributions are considered. The results are illustrated by estimating the stability of a simple autoregressive control process. Some examples of unstable processes are also provided.

Reviewer: Pavel Gapeev (London)

MSC:

60J05	Discrete-time Markov processes on general state spaces
93E10	Estimation and detection in stochastic control theory
90C40	Markov and semi-Markov decision processes
93D99	Stability of control systems

Keywords:

discrete-time Markov control process; average cost; contraction; stability inequality; Kantorovich metric

References:

[1]	Dynkin EB, Yushkevich AA (1979) Controlled Markov processes. Springer, New York
[2]	Gordienko E (1992) An estimate of the stability of optimal control of certain stochastic and deterministic systems. J Sov Math 59: 891–899 · Zbl 1267.49026 · doi:10.1007/BF01099115
[3]	Gordienko E, Hernández-Lerma O (1995) Average cost Markov control processes with weighted norms: existence of canonical policies. Appl Math 23: 199–218 · Zbl 0829.93067
[4]	Gordienko E, Lemus-Rodríguez E, Montes-de-Oca R (2008) Discounted cost optimality problem: stability with respect to weak metrics. Math Methods Oper Res (in press) · Zbl 1166.60041
[5]	Gordienko E, Salem F (2000) Estimates of Stability of Markov control processes with unbounded cost. Kybernetika 36: 195–210 · Zbl 1249.93176
[6]	Gordienko E, Yushkevich A (2003) Stability estimates in the problem of average optimal switching of a Markov chain. Math Methods Oper Res 57: 345–365 · Zbl 1116.90401
[7]	Hernández-Lerma O, Lasserre JB (1996) Discrete-time Markov control processes. Basic Optimality Criteria. Springer, New York
[8]	Montes-de-Oca R, Salem-Silva F (2005) Estimates for perturbations of average Markov decision processes with a minimal state and upper bounded by stochastically ordered Markov chains. Kybernetika 41: 757–772 · Zbl 1249.90313
[9]	Rachev ST, Rüschendorf L (1998) Mass transportation problems, vol II: Applications. Springer, New York
[10]	Van Dijk NM (1988) Perturbation theory for unbounded Markov reward processes with applications to queueing. Adv Appl Probab 20: 91–111 · Zbl 0642.60099
[11]	Van Dijk NM, Sladky K (1999) Error bounds for nonnegative dynamic models. J Optim Theory Appl 101: 449–474 · Zbl 0946.90106 · doi:10.1023/A:1021749829267
[12]	Vega-Amaya O (2003) The average cost optimality equation: a fixed point approach. Bol Soc Math Mexicana 9: 185–195 · Zbl 1176.90626