Continuous time Markov processes. An introduction.

*(English)*Zbl 1205.60002
Graduate Studies in Mathematics 113. Providence, RI: American Mathematical Society (AMS) (ISBN 978-0-8218-4949-1/hbk). xii, 271 p. (2010).

This is, as the title says, a textbook on the theory of continuous time Markov processes. Many important aspects of the theory are explained concisely and illustrated with a variety of examples. The author is always giving heuristic explanations in addition to the proofs, nothing is left unmotivated. The book can serve very well as a basis for a course on the subject, and one of the highlights is the big selection of well chosen exercises.

Before introducing the general theory of Markov processes, the example of Brownian motion is treated in some detail. Many important aspects of Markov processes are presented here, and the author often points out which properties of the Brownian motion were used in a proof, and to which kind of processes one could therefore generalize the result. The construction of a continuous version of Brownian motion is in the spirit of Kolmogorov’s continuity criterion and based on moment estimates. Proving the \(\alpha\)-Hölder continuity is left as an exercise, as well as giving another construction based on Haar functions. The Markov property of Brownian motion is shown and used for example to prove Blumenthal’s \(0-1\) law. Stopping times are introduced, the strong Markov property is shown, and again used to show many properties of Brownian motion, e.g. the reflection principle. There is a short introduction to martingales before the first chapter ends with a treatment of the Skorokhod embedding problem, which in turn is applied to give a very elegant proof of Donsker’s theorem.

The second chapter treats continuous time Markov processes on countable state spaces, i.e., continuous time Markov chains. The main intention of this chapter is to give the reader a feeling how to understand the generator of a Markov process (in this case the \(Q\)-matrix), and how one can go from the infinitesimal description of the transition laws to the semigroup and back. This is done in quite some detail, with a special focus on “degenerate” cases such as Blackwell’s example. Kolmogorov’s equations are introduced, and starting from the backward equation, a solution is constructed. This analytic construction is then given meaning with the probabilistic interpretation. Stability results of the Markov chain like existence of stationary measures, recurrence or irreducibility are related to the transition operator and the generator of the process. The chapter ends with some important examples of continuous time Markov chains and their properties.

Next there is an introduction to Feller processes. Here the student will profit from the intuition gained in the last chapter, to easier understand the heuristic meaning of a generator. It is pointed out that some important discrete examples from the last section are in fact not Feller processes. The author touches upon a variety of subjects: the Hille-Yosida theorem is proven, the concept of a core is introduced, and it is explained how one can see from the generator whether a given measure is stationary for a Feller process. There are short sections on martingale problems, duality, and the Feynman-Kac formula. As in the previous chapters, everything is always motivated. The Wright-Fisher diffusion is heuristically derived as the limit of a population model, and then rigorously defined using the Hille-Yosida theorem. The behavior near the boundary of the process then serves as transmission to the general study of how the boundary behavior of a Feller process is reflected in its generator.

The fourth chapter treats interacting particle systems. It is worked out very well how they relate to other Feller processes, and why they are different from most “classical” Markov processes. Special focus is put on the characterization of stationary measures. The first part is about the construction and some properties of general spin systems. Then there are two sections dedicated to the voter model and the contact process respectively, where these two are studied in more detail. Finally, an interacting particle system is treated that is not a spin system: the exclusion process.

Stochastic integration for continuous martingales is briefly introduced. Here the focus is more on a heuristic derivation of the most important concepts, and on giving an intuitive understanding, rather than on obtaining the greatest generality. Itô’s formula is shown, and the concept of local time is introduced. Stochastic differential equations in dimension one are treated with the help of scale functions and local time.

The last chapter treats the fruitful relation between Brownian motion and potential theory: Harmonic functions and the Dirichlet problem are introduced, and treated with elementary methods. Then some properties of multidimensional Brownian motion are developed, to then return to the Dirichlet problem and show that its solution is given by \( x \mapsto E^x (f(W(\tau_D))\) in the case of a bounded domain \(D\). This is generalized to unbounded domains. Finally, also the Poisson equation is given a probabilistic representation.

Before introducing the general theory of Markov processes, the example of Brownian motion is treated in some detail. Many important aspects of Markov processes are presented here, and the author often points out which properties of the Brownian motion were used in a proof, and to which kind of processes one could therefore generalize the result. The construction of a continuous version of Brownian motion is in the spirit of Kolmogorov’s continuity criterion and based on moment estimates. Proving the \(\alpha\)-Hölder continuity is left as an exercise, as well as giving another construction based on Haar functions. The Markov property of Brownian motion is shown and used for example to prove Blumenthal’s \(0-1\) law. Stopping times are introduced, the strong Markov property is shown, and again used to show many properties of Brownian motion, e.g. the reflection principle. There is a short introduction to martingales before the first chapter ends with a treatment of the Skorokhod embedding problem, which in turn is applied to give a very elegant proof of Donsker’s theorem.

The second chapter treats continuous time Markov processes on countable state spaces, i.e., continuous time Markov chains. The main intention of this chapter is to give the reader a feeling how to understand the generator of a Markov process (in this case the \(Q\)-matrix), and how one can go from the infinitesimal description of the transition laws to the semigroup and back. This is done in quite some detail, with a special focus on “degenerate” cases such as Blackwell’s example. Kolmogorov’s equations are introduced, and starting from the backward equation, a solution is constructed. This analytic construction is then given meaning with the probabilistic interpretation. Stability results of the Markov chain like existence of stationary measures, recurrence or irreducibility are related to the transition operator and the generator of the process. The chapter ends with some important examples of continuous time Markov chains and their properties.

Next there is an introduction to Feller processes. Here the student will profit from the intuition gained in the last chapter, to easier understand the heuristic meaning of a generator. It is pointed out that some important discrete examples from the last section are in fact not Feller processes. The author touches upon a variety of subjects: the Hille-Yosida theorem is proven, the concept of a core is introduced, and it is explained how one can see from the generator whether a given measure is stationary for a Feller process. There are short sections on martingale problems, duality, and the Feynman-Kac formula. As in the previous chapters, everything is always motivated. The Wright-Fisher diffusion is heuristically derived as the limit of a population model, and then rigorously defined using the Hille-Yosida theorem. The behavior near the boundary of the process then serves as transmission to the general study of how the boundary behavior of a Feller process is reflected in its generator.

The fourth chapter treats interacting particle systems. It is worked out very well how they relate to other Feller processes, and why they are different from most “classical” Markov processes. Special focus is put on the characterization of stationary measures. The first part is about the construction and some properties of general spin systems. Then there are two sections dedicated to the voter model and the contact process respectively, where these two are studied in more detail. Finally, an interacting particle system is treated that is not a spin system: the exclusion process.

Stochastic integration for continuous martingales is briefly introduced. Here the focus is more on a heuristic derivation of the most important concepts, and on giving an intuitive understanding, rather than on obtaining the greatest generality. Itô’s formula is shown, and the concept of local time is introduced. Stochastic differential equations in dimension one are treated with the help of scale functions and local time.

The last chapter treats the fruitful relation between Brownian motion and potential theory: Harmonic functions and the Dirichlet problem are introduced, and treated with elementary methods. Then some properties of multidimensional Brownian motion are developed, to then return to the Dirichlet problem and show that its solution is given by \( x \mapsto E^x (f(W(\tau_D))\) in the case of a bounded domain \(D\). This is generalized to unbounded domains. Finally, also the Poisson equation is given a probabilistic representation.

Reviewer: Nicolas Perkowski (Berlin)

##### MSC:

60-01 | Introductory exposition (textbooks, tutorial papers, etc.) pertaining to probability theory |

60J25 | Continuous-time Markov processes on general state spaces |

60J27 | Continuous-time Markov processes on discrete state spaces |

60J28 | Applications of continuous-time Markov processes on discrete state spaces |

60J35 | Transition functions, generators and resolvents |

60J45 | Probabilistic potential theory |

60J55 | Local time and additive functionals |

60J60 | Diffusion processes |

60J65 | Brownian motion |