The expected discounted reward from a Markov replacement process. (English) Zbl 0616.60089

Let \(\{X_ t\), \(t\geq 0\}\) be a homogeneous Markov process with state space \(I=\{1,...,r\}\). Its development can be influenced by the controller’s action called replacement. A stationary replacement policy f is the prescription to realize instantaneously the replacement \(j\to f(j)\) whenever the transition in state \(j\in I_ f\subset I\) occurs. No replacements are made in states \(j\not\in I_ f\). \(E^ f_ j\) denotes the mathematical expectation in a replacement process under the stationary replacement policy f and under the condition j being the initial state.
Let \(R_ T\) be the reward from the process up to the time T, \(R=\int^{\infty}_{0}e^{-\lambda T}dR(T)\), \(\lambda >0\), the discounting of the reward. The article derives a system of equations for establishing the expected discounted reward \(E^ f_ jR=\vartheta_ f(j)\), \(j=1,...,r\), and the uniqueness of its solution is proved.
R. A. Howard’s iteration procedure [Dynamic programming and Markov processes. (1960; Zbl 0091.16001)] for finding the maximal expected discounted reward \[ {\hat \vartheta}(j)=\max_{f}\{\vartheta_ f(j): f\text{ stationary replacement policy\}, }j\in I, \] and the responsive optimal stationary replacement policy is described.


60K20 Applications of Markov renewal processes (reliability, queueing networks, etc.)


Zbl 0091.16001
Full Text: EuDML


[1] Howard R. A.: Dynamic programming and Markov Processes. M. I. T. Press and John Wiley, New York-London (1960). · Zbl 0091.16001
[2] Mandl P.: An identity for Markovian replacement processes. J. Appl. Prob. 6, No. 2, 348-354 (1969). · Zbl 0192.55203
[3] Mandl P.: Řízené Markovovy řetězce. příloha časopisu Kybernetika, roč. 5, Academia Praha (1969).
[4] Sarymsakov T. A.: Osnovy teorii processov Markova. Moskva (1954).
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.