# zbMATH — the first resource for mathematics

Long-term average cost control problems for continuous time Markov processes: A survey. (English) Zbl 0531.93068
This paper is a survey of long term average cost control problems, namely the cost is defined by $J(v)=\underline{\lim}_{T\to \infty}\frac{1}{T}E\int^{T}_{0}f(y(s,v),v(s))ds,$ where $$y(\cdot,v)$$ is the response for control v. The object is to minimize J(v). The following simple example illustrate the kind of phenomena treated in the paper. Example: Let V be the totality of Lipschitz continuous functions. For $$v\in V$$, $$y(t,v)$$ is the solution of SDE, $dy_ x(t)=v(y_ x(t))dt+\sqrt{2}dw(t),\quad y_ x(0)=x(\in R^ 1).$ The discounted cost is given by $$J_ x^{\alpha}(v)=E\int^{\infty}_{0}e^{-\alpha t}(y_ x(t)^ 2+v^ 2(y_ x(t)))dt$$. The optimal cost function $$u_{\alpha}(x)=\inf_{v\in V}J_ x^{\alpha}(v)$$ satisfies the Hamilton-Jacobi-Bellman (H-J-B) equation $$u_{\alpha}''(x)-\inf_{v\in R^ 1}(vu'_{\alpha}(x)+v^ 2)+\alpha u_{\alpha}=x^ 2$$. Actually $$u_{\alpha}(x)=(\sqrt{\alpha^ 2+4}-\alpha)(\frac{x^ 2}{2}+\frac{1}{\alpha})$$. Hence, $$\lim_{\alpha \to 0}\alpha u_{\alpha}(x)=2$$ and $$W(x)=\lim_{\alpha \to 0}(u_{\alpha}(x)- u_{\alpha}(0))=x^ 2$$ satisfies $-W''(x)-\inf_{v\in R^ 1}(vW'(x)+v^ 2)+\lambda =x^ 2.$ Moreover, putting $$V_ 0=\{v\in V;\lim_{T\to \infty}\frac{1}{T}EW(y_ x(T,v))=0\}$$, $\lambda =\inf_{v\in V_ 0}(\lim_{T\to \infty}\frac{1}{T}E\int^{T}_{0}y^ 2_ x(t)+v^ 2(y_ x(t))dt)$ and an optimal control $$\hat v(x)=-x$$ is a minimum selection of $$(vW'(x)+v^ 2)$$. In many cases, the problem relates to a solution $$(\lambda,W)$$ of the H-J-B equation $$\inf_{v\in K}(A^ vW+f(x,v))- \lambda =0,$$ and using the discounted problem a solution is obtained. The author treats three problems: continuous control, stopping problem and impulse control, and discusses some open problems.
Reviewer: M.Nisio

##### MSC:
 93E20 Optimal stochastic control 60J25 Continuous-time Markov processes on general state spaces 49J55 Existence of optimal solutions to problems involving randomness 49J40 Variational inequalities 49K45 Optimality conditions for problems involving randomness 60G40 Stopping times; optimal stopping problems; gambling theory 49L20 Dynamic programming in optimal control and differential games 62L15 Optimal stopping in statistics 90C39 Dynamic programming
Full Text:
##### References:
 [1] Bellman R.: J. Math. Mech. 6 (1957), 679-684. [2] Bensoussan A.: Stochastic Control by Functional Analysis Methods, North-Holland, Amsterdam, 1982. · Zbl 0474.93002 [3] Bensoussan A. and Lions J. L.: Applications des inéquations variationnelles en contrôle stochastique, Dunod, Paris, 1978. · Zbl 0411.49002 [4] Bensoussan A. and Lions J. L.: Contrôle impulsionnel et inéquations quasi-variationnelles, Dunod, Paris, 1982. · Zbl 0491.93002 [5] Bensoussan A. and Lions J. L.: ?Nouvelles formulations de problèmes de contrôle impulsionnel et applications?, CRAS, Paris, 1973, pp. 1189-1192. · Zbl 0266.49007 [6] Bensoussan A. and Lions J. L.: in Theory of Nonlinear Operators, Akademic Verlag, Berlin, 1978. · Zbl 0437.49014 [7] Bismut J. M. and Skalli B.: Z. Wahz. Verw. G. 39 (1977), 301-313. · Zbl 0342.60036 · doi:10.1007/BF01877497 [8] Blackwell D. Ann. Math. Stat. 33 (1962), 719-726. · Zbl 0133.12906 · doi:10.1214/aoms/1177704593 [9] Doshi B. T.: Stoch. Proc. Appl. 4 (1976), 55-77. · Zbl 0322.60069 · doi:10.1016/0304-4149(76)90026-0 [10] Dynkin E. B.: Markov Processes, Vols. I, II, Springer-Verlag, Heidelberg, 1965. · Zbl 0132.37901 [11] Dynkin E. B.: Dokl. Acad. Nauk USSR, 150 (1963), 238-240. [12] Dynkin E. B. and Yuskevich A. A.: Controlled Markov Processes, Springer-Verlag, Berlin, 1979. [13] El Karoui, N.: Mèthodes probabilistes en contrôle stochastique, to be published. [14] Fayolle G. and Robin M.: in Modeling and Performance Evaluation of Computer Systems, North-Holland, Amsterdam, 1976. [15] Federgruen A., Hordijk A. and Tijms H. C.: Stoch. Proc. Appl. 9 (1979), 223-235. · Zbl 0422.90084 · doi:10.1016/0304-4149(79)90034-6 [16] Fleming W. and Rishel R.: Optimal Deterministic and Stochastic Control, Springer-Verlag, Berlin, 1975. · Zbl 0323.49001 [17] Has’minskii, R. Z.: Stochastic Stability of Differential Equations, Sijthoff & Noordhoff, 1980. [18] Hordijk, A.: ?Dynamic Programming and Markov Potential Theory?, Mathematical Center Tr. No. 51, Amsterdam, 1974. · Zbl 0284.49012 [19] Howard R. A.: Dynamic Programming and Markov Processes, MIT Press. Cambridge, Mass, 1960. · Zbl 0091.16001 [20] Kakumanu P. K.: Siam J. Control 10 (1972), 210-220. · Zbl 0271.60066 · doi:10.1137/0310016 [21] Karatzas, J.: ?A Class of Singular Stochastic Control Problem?, to appear. · Zbl 0534.93065 [22] Kogan Y. A.: Theor. Proba. Applic. 14 (1969), 496-502. · Zbl 0281.60090 · doi:10.1137/1114063 [23] Lanery, E.: ?Etude asymptotique des systèmes markoviens à commande?, Revue inf. Recherche opérationnelle lère annee, No. 5, pp. 3-56. · Zbl 0159.48804 [24] Lasry, J. M.: ?Contrôle stochastique ergodique?, Thèse Université de Paris IX, 1974. [25] Lepeltier, J. P. and Marchal, B.: ?Théorie générale du contrôle impulsionnel?, Thèse, 1980. [26] Lippman S. A.: Ann. Math. Stat. 42 (1971), 1717-1726. · Zbl 0231.90057 · doi:10.1214/aoms/1177693170 [27] Lorenzani, M.: ?Optimal Stopping for Markov Processes?, Note. Rend. Cont. Acad. Sci., Rome, 1981. · Zbl 0513.60046 [28] Mandl P.: Analytic Treatment of One-Dimensional Markov Processes, Springer-Verlag, New York, 1968. · Zbl 0179.47802 [29] Makowski, A.: ?Local Optimality Conditions for Optimal Stopping?, CSR-81-23/1981, Dept of Elect. Eng. University of Maryland. [30] Menaldi, J. L.: ?On the Degenerate variational Inequality with Neumann Boundary Conditions, Siam J. Control Opt., to be published. · Zbl 0452.49017 [31] Menaldi J. L.: Siam J. Control Opt. 18 (1980), 697-721. · Zbl 0462.93045 · doi:10.1137/0318052 [32] Miller B.: J. Appl. Prob. 18 (1981), 361-377. · Zbl 0464.60046 · doi:10.2307/3213283 [33] Prabhu N. and Stidham S.: in Mathematical Methods in Queueing Theory, (Lecture Notes in Economics and Mathematical Systems, 98) Springer-Verlag, Heidelberg, 1974. · Zbl 0285.60087 [34] Puterman, M.: ?On the Optimal Control of Diffusion Processes?, TR No. 24, Stanford University, 1972. · Zbl 0336.93047 [35] Robin, M.: ?Contrôle impulsionnel des processus de Markov?, Thesis, University of Paris IX, 1978. · Zbl 0397.93039 [36] Robin M.: Siam J. Control Opt. 19 (1981), 333-358. · Zbl 0461.93062 · doi:10.1137/0319020 [37] Robin M.: in System Modeling and Optimization. (Lecture Notes in Control and Information Sciences, 38), Springer-Verlag, Heidelberg, 1982. [38] Robin, M.: ?Asymptotics in Quasi-Variational Inequalities and Ergodic Control Problems?, to be published, Systems and Control Letters 1983. · Zbl 0517.93066 [39] Rosberg Z., Varayia P. and Walrand J.: IEEE Tr. Auto-Control, AC 27 (1982), 600-610. · Zbl 0497.90024 · doi:10.1109/TAC.1982.1102957 [40] Ross S. M.: Ann. Math. Stat. 42 (1971), 1767-1768. · Zbl 0238.62094 · doi:10.1214/aoms/1177693180 [41] Ross S. M.: J. Appl. Prob. 7 (1970), 649-656. · Zbl 0204.51704 · doi:10.2307/3211944 [42] Ross S. M.: Applied Probability Models with Optimization Applications, Holden Day, San Francisco, 1970. · Zbl 0213.19101 [43] Schweitzer P.: ?A Non-Expansive Mapping Approach to Undiscounted Semi-Markovian Decision Processes?, WP. 8109, 1981, Graduate School of Management, University of Rochester, U.S.A. [44] Stroock D. and Varadhan S.: Comm. Pure Appl. Math. 24 (1971), 147-225. · Zbl 0227.76131 · doi:10.1002/cpa.3160240206 [45] Sulem, A.: to be published. [46] Stettner, L.: ?On Impulsive Control with Long-Run Average Cost Criterion?, to appear in Studia Mathematica. · Zbl 0509.93069 [47] Shiryaev A. N.: Optimal Stopping Rules, Springer-Verlag, New York, 1978. · Zbl 0391.60002 [48] Tarres, R.: ?Comportement asymptotique d’un problème de contrôle stochastique?, Cahier de math. de la Décision No. 8215, Université de Paris IX, 1982. [49] Vanderduyn-Shouten F. A. and Hordijk A.: ?On the Existence of Average Optimal Policies in Markov Decision Drift Processes with General State and Action Space?, Vrije Universiteit, Amsterdam, TR No. 73, 1981. [50] Wijngaard J.: Math. Oper. Res. 2 (1977), 91-102. · Zbl 0395.90081 · doi:10.1287/moor.2.1.91 [51] Yu Chung Liao- ?On Optimal Control of a Brownian Motion and Approximation of Queueing Processes?, PhD Thesis, Brown University, R.I. 1982. [52] Yuskevich A. A.: Theor. Proba. Applic. 26 (1981); 796-802. · Zbl 0499.60094 · doi:10.1137/1126085 [53] Zabczyk J.: in Stochastic Control Theory and Stochastic Differential Systems (Lecture Notes in Control and Information Sciences 16), Springer-Verlag, Heidelberg, 1979. · Zbl 0435.93047
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.