zbMATH — the first resource for mathematics

Discrete-time Markov control processes. Basic optimality criteria. (English) Zbl 0840.93001
Applications of Mathematics. 30. New York, NY: Springer-Verlag. xiv, 216 p. (1995).
This advanced book is the first volume of a planned two-volume series of the authors devoted to the theory of discrete-time Markov control processes (MCPs) with a semicontinuous-semicompact model. It is organized in six chapters, five appendices and many references. Chapter 1 is an informal introduction to MCPs, presenting basic concepts and preliminary examples. In Chapter 2 the concepts of MCP and Markov control policies are rigorously described in mathematical terms. Chapter 3 deals with finite horizon problems, in which the basic theorem in the theory of MCPs, i.e., the dynamic programming (DP) theorem is proved and the conditions for the so-called semicontinuous-semicompact model which yield the existence of optimal policies for the problems are proposed. Chapter 4 deals with the problems of minimization of the infinite-horizon \(\alpha\)-discounted cost (\(\alpha\)-DC) criterion, including the proof of the corresponding DP theorem, policy iteration and other approximations, optimality criteria and asymptotic optimality criteria etc\(\dots\) Chapter 5 deals with the problems of minimization of the average cost (AC) criterion, in which the existence of AC-optimal policies is studied, the vanishing discount approach is informally discussed, and the value iteration procedure is introduced. Chapter 6 presents the infinite-dimensional linear programming formulation of MCPs under the \(\alpha\)-DC criterion as well as under the AC criterion.
With the appendices for the prerequisites, the book is almost self-contained. The whole text reflects the most recent progress in the field of MCPs, in which many contributions are made by the authors themselves. Although the book is virtually an advanced mathematical monograph, the problems, the models and many examples discussed here are also very useful for engineering, economics, population processes, management science etc\(\dots\).

93-01 Introductory exposition (textbooks, tutorial papers, etc.) pertaining to systems and control theory
93E20 Optimal stochastic control
90C40 Markov and semi-Markov decision processes