Ergodicity, stabilization, and singular perturbations for Bellman-Isaacs equations.

*(English)*Zbl 1209.35001
Mem. Am. Math. Soc. 960, v, 77 p. (2010).

The controlled system with a small parameter \(\varepsilon > 0\)

\[ \begin{alignedat}{2} dx_s&= f(x_s,y_s,\alpha_s)ds+\sigma(x_s,y_s,\alpha_s)dW_s,&\quad &x_0\in\mathbb{R}^n,\\ dy_s&=\frac{1}{\varepsilon} g(x_s,y_s,\alpha_s)ds+\frac{1}{\sqrt{\varepsilon}}\tau(x_s,y_s,\alpha_s)dW_s,&\quad &y_0\in \mathbb{R}^m,\end{alignedat}\tag{1} \] is considered where \(W_s\) is a \(r\)-dimensional Brownian motion, and the optimal control problem of minimizing the cost functional

\[ J(t,x,y,\alpha)=E_{(x,y)}\biggl[\int_{0}^{t}l(x_s,y_s,\alpha_s)ds+h(x_t,y_t)\biggr], \] as \(\alpha\) varies in the set of admissible control functions \({\mathcal A}(t)\). This is a model of systems where some state variables, \(y_s\) here, evolve at a much faster time scale than the other variables, \(x_s\). Passing to the limit as \(\varepsilon \rightarrow 0_+\) is a classical singular perturbation problem. Its solution leads to the elimination of the state variables \(y\) and the reduction of the dimension of the system from \(n+m\) to \(n\). Of course the limit control problem keeps some informations on the fast part of the system.

There is a large mathematical and engineering literature on singular perturbation problems in control, both in the deterministic \((\sigma \equiv 0\), \(\tau \equiv 0)\) and in the stochastic case. The authors begin with the methods that aim at deriving directly an explicit description of the limit system. The first approach is the order reduction method originated in the work of Levinson and Tikhonov on ODEs and extended to deterministic control systems. For deterministic systems with more general asymptotic behavior of the fast variables the classical averaging method for ODEs of Krylov and Bogolyubov was developed to the theory of limit occupational measures for control systems by others authors. Stochastic systems with uncontrolled fast dynamics \((g=g(x,y),~\tau=\tau(x,y))\) were studied by Bensoussan.

The controlled case appears much more difficult and some results were obtained only in last ten years. A different approach to the singular perturbation problem consists of studying the limit as \(\varepsilon \rightarrow 0_+\) of the value function

\[ u^\varepsilon(t,x,y)= \inf_{\alpha\in{\mathcal A}(t)} J(t,x,y,\alpha) \] and characterizing it as the unique solution of a limiting Hamilton-Jacobi-Bellman equation (HJB). This approach starts from the HJB equation in \(\mathbb{R}^{n+m}\) satisfied by \(u^\varepsilon\), that in the deterministic case is of first order

\[ u_t^\varepsilon+ \max_{\alpha\in{\mathcal A}(t)}\bigg(-f(x,y,\alpha)\cdot D_xu^\varepsilon- g(x,y,\alpha)\cdot\frac{D_yu^\varepsilon}{\varepsilon}- l(x,y,\alpha)\bigg)=0, \] and in the stochastic case is of second order

\[ u_t^\varepsilon+ \max_{\alpha\in{\mathcal A}(t)} L^\alpha \bigg(x,y,D_xu^\varepsilon,\frac{D_yu^\varepsilon}{\varepsilon}, D_{xx}u^\varepsilon,\frac{D_{yy}u^\varepsilon}{\varepsilon},\frac{D_{xy}u^\varepsilon}{\sqrt{\varepsilon}}\bigg)=0, \] where \(L^\alpha\) is the generator of the process in (1) with the constant control \(\alpha\) and \(\varepsilon=1\). One expects that the limit \(u(t,x)\) does not depend on \(y\) and solves a PDE in \(\mathbb{R}^n\) governed by an effective Hamiltonian \(\overline{H}\). It turns out that \(\overline{H}\) is the value of an ergodic control problem in \(\mathbb{R}^m\) for the fast subsystem with frozen slow variable \(x\) and \(\varepsilon=1\). Once this is found, one tries to prove that the limit of \(u^\varepsilon\) solves the effective PDE

\[ u_t +\overline{H}(x,D_xu,D_{xx}u)=0. \] If this PDE, with suitable initial conditions, has at most one solution, then we have a characterization of the limit \(u(t,x)\) and a way to compute it, at least in principle, by solving a lower dimensional PDE. The theory of viscosity solutions for first order and for second order, degenerate parabolic, fully nonlinear equations is the natural framework for this approach.

The purpose of the present paper is to provide a reference framework for the study of singular perturbations with PDE methods in the generality of stochastic differential games, by complementing the abstract theory with several sets of conditions that make it work successfully. The main part concerns the properties of ergodicity and stabilization.

Let us point out the main additions that this paper makes to the existing literature. First of all it gives a general unified method for studying singular perturbations for deterministic and stochastic systems, and for one as well as two completing controllers. Usually the assumptions and the methods are quite different in the deterministic and the stochastic setting.

Ergodic control has independent interest and a large literature. The contribution of the authors is essentially in the extension from the case of a single player to games.

The results about stabilization are entirely new, although the methods are inspired by those employed for ergodicity.

Finally, let us mention that the authors extended some results of this paper to problems with an arbitrary number of scales.

\[ \begin{alignedat}{2} dx_s&= f(x_s,y_s,\alpha_s)ds+\sigma(x_s,y_s,\alpha_s)dW_s,&\quad &x_0\in\mathbb{R}^n,\\ dy_s&=\frac{1}{\varepsilon} g(x_s,y_s,\alpha_s)ds+\frac{1}{\sqrt{\varepsilon}}\tau(x_s,y_s,\alpha_s)dW_s,&\quad &y_0\in \mathbb{R}^m,\end{alignedat}\tag{1} \] is considered where \(W_s\) is a \(r\)-dimensional Brownian motion, and the optimal control problem of minimizing the cost functional

\[ J(t,x,y,\alpha)=E_{(x,y)}\biggl[\int_{0}^{t}l(x_s,y_s,\alpha_s)ds+h(x_t,y_t)\biggr], \] as \(\alpha\) varies in the set of admissible control functions \({\mathcal A}(t)\). This is a model of systems where some state variables, \(y_s\) here, evolve at a much faster time scale than the other variables, \(x_s\). Passing to the limit as \(\varepsilon \rightarrow 0_+\) is a classical singular perturbation problem. Its solution leads to the elimination of the state variables \(y\) and the reduction of the dimension of the system from \(n+m\) to \(n\). Of course the limit control problem keeps some informations on the fast part of the system.

There is a large mathematical and engineering literature on singular perturbation problems in control, both in the deterministic \((\sigma \equiv 0\), \(\tau \equiv 0)\) and in the stochastic case. The authors begin with the methods that aim at deriving directly an explicit description of the limit system. The first approach is the order reduction method originated in the work of Levinson and Tikhonov on ODEs and extended to deterministic control systems. For deterministic systems with more general asymptotic behavior of the fast variables the classical averaging method for ODEs of Krylov and Bogolyubov was developed to the theory of limit occupational measures for control systems by others authors. Stochastic systems with uncontrolled fast dynamics \((g=g(x,y),~\tau=\tau(x,y))\) were studied by Bensoussan.

The controlled case appears much more difficult and some results were obtained only in last ten years. A different approach to the singular perturbation problem consists of studying the limit as \(\varepsilon \rightarrow 0_+\) of the value function

\[ u^\varepsilon(t,x,y)= \inf_{\alpha\in{\mathcal A}(t)} J(t,x,y,\alpha) \] and characterizing it as the unique solution of a limiting Hamilton-Jacobi-Bellman equation (HJB). This approach starts from the HJB equation in \(\mathbb{R}^{n+m}\) satisfied by \(u^\varepsilon\), that in the deterministic case is of first order

\[ u_t^\varepsilon+ \max_{\alpha\in{\mathcal A}(t)}\bigg(-f(x,y,\alpha)\cdot D_xu^\varepsilon- g(x,y,\alpha)\cdot\frac{D_yu^\varepsilon}{\varepsilon}- l(x,y,\alpha)\bigg)=0, \] and in the stochastic case is of second order

\[ u_t^\varepsilon+ \max_{\alpha\in{\mathcal A}(t)} L^\alpha \bigg(x,y,D_xu^\varepsilon,\frac{D_yu^\varepsilon}{\varepsilon}, D_{xx}u^\varepsilon,\frac{D_{yy}u^\varepsilon}{\varepsilon},\frac{D_{xy}u^\varepsilon}{\sqrt{\varepsilon}}\bigg)=0, \] where \(L^\alpha\) is the generator of the process in (1) with the constant control \(\alpha\) and \(\varepsilon=1\). One expects that the limit \(u(t,x)\) does not depend on \(y\) and solves a PDE in \(\mathbb{R}^n\) governed by an effective Hamiltonian \(\overline{H}\). It turns out that \(\overline{H}\) is the value of an ergodic control problem in \(\mathbb{R}^m\) for the fast subsystem with frozen slow variable \(x\) and \(\varepsilon=1\). Once this is found, one tries to prove that the limit of \(u^\varepsilon\) solves the effective PDE

\[ u_t +\overline{H}(x,D_xu,D_{xx}u)=0. \] If this PDE, with suitable initial conditions, has at most one solution, then we have a characterization of the limit \(u(t,x)\) and a way to compute it, at least in principle, by solving a lower dimensional PDE. The theory of viscosity solutions for first order and for second order, degenerate parabolic, fully nonlinear equations is the natural framework for this approach.

The purpose of the present paper is to provide a reference framework for the study of singular perturbations with PDE methods in the generality of stochastic differential games, by complementing the abstract theory with several sets of conditions that make it work successfully. The main part concerns the properties of ergodicity and stabilization.

Let us point out the main additions that this paper makes to the existing literature. First of all it gives a general unified method for studying singular perturbations for deterministic and stochastic systems, and for one as well as two completing controllers. Usually the assumptions and the methods are quite different in the deterministic and the stochastic setting.

Ergodic control has independent interest and a large literature. The contribution of the authors is essentially in the extension from the case of a single player to games.

The results about stabilization are entirely new, although the methods are inspired by those employed for ergodicity.

Finally, let us mention that the authors extended some results of this paper to problems with an arbitrary number of scales.

Reviewer: Vasile Iftode (Bucureşti)

##### MSC:

35-02 | Research exposition (monographs, survey articles) pertaining to partial differential equations |

35F21 | Hamilton-Jacobi equations |

35B25 | Singular perturbations in context of PDEs |

35B27 | Homogenization in context of PDEs; PDEs in media with periodic structure |

35Kxx | Parabolic equations and parabolic systems |

93C70 | Time-scale analysis and singular perturbations in control/observation systems |

49N70 | Differential games and control |

49L25 | Viscosity solutions to Hamilton-Jacobi equations in optimal control and differential games |

60J60 | Diffusion processes |

91A23 | Differential games (aspects of game theory) |

93E20 | Optimal stochastic control |