Estimating the probability that a simulated system will be the best.

*(English)*Zbl 0993.62019Summary: Consider a stochastic simulation experiment consisting of \(v\) independent vector replications consisting of an observation from each of k independent systems. Typical system comparisons are based on mean (long-run) performance. However, the probability that a system will actually be the best is sometimes more relevant, and can provide a very different perspective than the systems’ means. Empirically, we select one system as the best performer (i.e., it wins) on each replication. Each system has an unknown constant probability of winning on any replication and the numbers of wins for the individual systems follow a multinomial distribution.

Procedures exist for selecting the system with the largest probability of being the best. This paper addresses the companion problem of estimating the probability that each system will be the best. The maximum likelihood estimators (MLEs) of the multinomial cell probabilities for a set of \(v\) vector replications across \(k\) systems are well known. We use these same \(v\) vector replications to form \(v^k\) unique vectors (termed pseudo-replications) that contain one observation from each system and develop estimators based on AVC (All Vector Comparisons). In other words, we compare every observation from each system with every combination of observations from the remaining systems and note the best performer in each pseudo-replication. AVC provides lower variance estimators of the probability that each system will be the best than the MLEs. We also derive confidence intervals for the AVC point estimators, present a portion of an extensive empirical evaluation and provide a realistic example.

Procedures exist for selecting the system with the largest probability of being the best. This paper addresses the companion problem of estimating the probability that each system will be the best. The maximum likelihood estimators (MLEs) of the multinomial cell probabilities for a set of \(v\) vector replications across \(k\) systems are well known. We use these same \(v\) vector replications to form \(v^k\) unique vectors (termed pseudo-replications) that contain one observation from each system and develop estimators based on AVC (All Vector Comparisons). In other words, we compare every observation from each system with every combination of observations from the remaining systems and note the best performer in each pseudo-replication. AVC provides lower variance estimators of the probability that each system will be the best than the MLEs. We also derive confidence intervals for the AVC point estimators, present a portion of an extensive empirical evaluation and provide a realistic example.

##### MSC:

62F10 | Point estimation |

62F12 | Asymptotic properties of parametric estimators |

62G05 | Nonparametric estimation |

PDF
BibTeX
XML
Cite

\textit{J. O. Miller} et al., Nav. Res. Logist. 49, No. 4, 341--358 (2002; Zbl 0993.62019)

Full Text:
DOI

##### References:

[1] | Bechhofer, Ann Math Stat 30 pp 102– (1959) · Zbl 0218.62064 · doi:10.1214/aoms/1177706362 |

[2] | Bechhofer, Ann Math Stat 29 pp 325– (1958) · doi:10.1214/aoms/1177706739 |

[3] | and Statistical inference, Pacific Grove, California, 1990. |

[4] | A multinomial ranking and selection procedure: Simulation and applications, Proceedings of the 1984 Winter Simulation Conference, S. Shepard, U.W. Pooch, and C.D. Pegden (Editors), The Institute of Electrical and Electronics Engineers, Piscataway, NJ, 1984, pp. 259-264. |

[5] | On selecting the best of K systems: An expository survey of indifference-zone multinomial procedures, Proceedings of the 1984 Winter Simulation Conference, S. Shepard, U.W. Pooch, and C.D. Pegden (Editors), The Institute of Electrical and Electronics Engineers, Piscataway, NJ, 1984, pp. 107-112. |

[6] | and Methods for selection the best system, Proceedings of the 1991 Winter Simulation Conference, B. Nelson, W.D. Kelton, and G. Clark (Editors), The Institute of Electrical and Electronics Engineers, Piscataway, NJ, 1991, pp. 177-186. |

[7] | personal communication, 1995. |

[8] | Lehmann, Ann Math Stat 34 pp 957– (1963) · Zbl 0203.21106 · doi:10.1214/aoms/1177704018 |

[9] | Efficient multinomial selection in simulation, unpublished Ph.D. Thesis, Department of Industrial, Welding, and Systems Engineering, The Ohio State University, Columbus, 1997. |

[10] | and How common random numbers affect multinomial selection, Proceedings of the 1997 Winter Simulation Conference, S. Andradottir, K.J. Healy, D.H. Withers, and B.L. Nelson, The Institute of Electrical and Electronics Engineers, Piscataway, NJ, 1997, pp. 342-347. |

[11] | Miller, Nav Res Logistics 45 pp 459– (1998) · Zbl 0940.90062 · doi:10.1002/(SICI)1520-6750(199808)45:5<459::AID-NAV2>3.0.CO;2-2 |

[12] | and Numerical recipes: The art of scientific computing (FORTRAN version), Cambridge University Press, Cambridge, 1989. |

[13] | and Introduction to the theory of nonparametric statistics, Wiley, New York, 1979. |

This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.