zbMATH — the first resource for mathematics

Monotonicity in a Markov decision process. (English) Zbl 0646.90088
Consider the Markov decision process with state space \(I=\{0,...,n\}\) and closed action set \(A\subseteq [0,\infty]\), having optimality equation \[ V_ i=\min_{a}\{a+c+\int^{i}_{j=0}p(i,j,a)V_ j\},\quad 1\leq i\leq n,\quad V_ 0=0, \] in which the transition probabilities are given by \[ p(i,j,a)=\left( \begin{matrix} i\\ j\end{matrix} \right)p^{aj}(1-p^ a)^{i-j}, \] 0\(<p<1\) and \(c>0\). The model can be interpreted in terms of collective sequential testing of n subjects. When \(A=[0,\infty]\) it is shown that \(V_ i\) is increasing and concave in i, and that the optimal actions \(a_ i\) are strictly increasing in i. However, when A is the set of positive integers, whilst the same properties hold for \(V_ i\), monotonicity of \(a_ i\) is more difficult to establish: a restriction on the values of the parameters p and c is given as a sufficient condition.
Reviewer: J.Preater

90C40 Markov and semi-Markov decision processes
Full Text: DOI