×

zbMATH — the first resource for mathematics

A methodology for fitting and validating metamodels in simulation. (English) Zbl 0985.65007
Summary: This paper proposes a methodology that replaces the usual ad hoc approach to metamodeling. This methodology considers validation of a metamodel with respect to both the underlying simulation model and the problem entity. It distinguishes between fitting and validating a metamodel, and covers four types of goal: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. The methodology consists of a metamodeling process with 10 steps. This process includes classic design of experiments (DOE) and measuring fit through standard measures such as \(R\)-square and cross-validation statistics. The paper extends this DOE to stagewise DOE, and discusses several validation criteria, measures, and estimators. The methodology covers metamodels in general (including neural networks); it also gives a specific procedure for developing linear regression (including polynomial) metamodels for random simulation.

MSC:
65C60 Computational problems in statistics (MSC2010)
62K05 Optimal statistical designs
62J05 Linear regression; mixed models
65D10 Numerical smoothing, curve fitting
Software:
bootstrap
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Alexopoulos, Seila, 1998. Output data analysis. In: Banks, J. (Ed.), Handbook of Simulation. Wiley, New York
[2] Balci, O.; Sargent, R.G., A methodology for cost-risk analysis in the statistical validation of simulation models, Communications of the ACM, 24, 4, 190-197, (1981)
[3] Barton, R.R., 1993. New tools for simulation metamodels. IMSE Working Paper 93-110, Department of Industrial and Management Systems Engineering, Penn State University, University Park, PA 16802
[4] Barton, R.R., 1994. Metamodeling: A state of the art review. In: Tew, J.D., Manivannan, S., Sadowski, Seila, A.F., (Eds.), Proceedings of the 1994 Winter Simulation Conference, pp. 237-244
[5] Beck, M.B., Ravetz, J.R., Mulkey, L.A., Barnwell, T.O., 1997. On the problem of model validation for predictive exposure assessments. Stochastic Hydrology and Hydraulics 11, pp. 229-254
[6] Bettonvil, B.; Kleijnen, J.P.C., Searching for important factors in simulation models with many factors: sequential bifurcation, European journal of operational research, 96, 1, 180-194, (1997) · Zbl 0924.90047
[7] Chen, B., 1985. A statistical validation procedure for discrete event simulation over experimental regions. Ph.D. Dissertation, Department of Industrial Engineering and Operations Research, Syracuse University, Syracuse, NY
[8] Cheng, R.C.H., Kleijnen, J.P.C., 1998. Improved designs of queuing simulation experiments with highly heteroscedastic responses. Operations Research (forthcoming)
[9] Diebold, F.X., Mariano, R.S., 1995. Comparing predictive accuracy. Journal of Business and Economic Statistics 13 (3), 253-263
[10] Donohue, J.M., 1995. The use of variance reduction techniques in the estimation of simulation metamodels. In: Alexopoulos, C., Kang, K., Lilegdon, W.R., Goldsman, D., (Eds.), Proceedings of the 1995 Winter Simulation Conference, pp. 195-199
[11] Efron, B., Tibshirani, T.J., 1993. Introduction to the Bootstrap. Chapman and Hall, New York · Zbl 0835.62038
[12] Ehrman, C.M.; Hamburg, M.; Krieger, A.M., A method for selecting a subset of alternatives for future decision making, European journal of operational research, 96, 407-416, (1996) · Zbl 0917.90009
[13] Friedman, L.W., 1996. The Simulation Metamodel. Kluwer, Dordrecht, The Netherlands
[14] Fu, M.C., Optimization via simulation: A review, Annals of operations research, 53, 199-247, (1994) · Zbl 0833.90089
[15] Goldsman, D., Nelson, B.L., 1998. Comparing Systems via Simulation. In: Banks, J. (Ed.), Handbook of Simulation. Wiley, New York
[16] Huber, K.P., Berthold, M.R., Szczerbicka, H., 1996. Analysis of simulation models with fuzzy graph based metamodeling. Performance Evaluation 27/28, 473-490 · Zbl 0900.68460
[17] Khuri, A.I., 1996a. Analysis of multiresponse experiments: A review. In: Ghosh, S. (Ed.), Statistical Design and Analysis of Industrial Experiments, Marcel Dekker, New York, pp. 231-246
[18] Khuri, A.I., 1996b. Multiresponse surface methodology. In: Ghosh, S., Rao, C.R. (Eds.), Handbook of Statistics, vol. 13. Elsevier, Amsterdam, pp. 377-406 · Zbl 0911.62071
[19] Kleijnen, J.P.C., 1987. Statistical tools for simulation practitioners. Marcel Dekker, New York · Zbl 0629.62004
[20] Kleijnen, J.P.C., Regression metamodels for simulation with common random numbers: comparison of validation tests and confidence intervals, Management science, 38, 8, 1164-1185, (1992) · Zbl 0753.62038
[21] Kleijnen, J.P.C., Case study: statistical validation of simulation models, European journal of operational research, 87, 1, 21-34, (1995) · Zbl 0914.90191
[22] Kleijnen, J.P.C., Verification and validation of simulation models, European journal of operational research, 82, 1, 145-162, (1995) · Zbl 0905.90119
[23] Kleijnen, J.P.C., Sensitivity analysis and optimization of system dynamics models: regression analysis and statistical design of experiments, System dynamics review, 11, 4, 275-288, (1995)
[24] Kleijnen, J.P.C., 1999. Experimental design for sensitivity analysis, optimization, and validation of simulation models. In: Banks, J., (Ed.), Handbook of Simulation. Wiley, New York. (Preprint: CentER Discussion Paper, no. 9752.)
[25] Kleijnen, J.P.C., Gaury, E., 1998. Risk analysis of robust system design. In: Medeiros, D.J., Watson, E.F., Carson, J.S., Manivannan, M.S. (Eds.), Proceedings of the 1998 Winter Simulation Conference, pp. 1533-1540
[26] Kleijnen, J.P.C.; Standridge, C., Experimental design and regression analysis: an FMS case study, European journal of operational research, 33, 3, 257-261, (1988)
[27] Kleijnen, J.P.C., van Groenendaal, W., 1992. Simulation: A statistical perspective. Wiley, Chichester. · Zbl 0797.90001
[28] Kleijnen, J.P.C., Cheng, R.C.H., Bettonvil, B., 1998. Validation of trace-driven simulation models: bootstrapped tests. Working Paper
[29] Kleijnen, J.P.C., Feelders, A.J., Cheng, R.C.H., 1998. Bootstrapping and validation of metamodels in simulation. In: Medeiros, D.J., Watson, E.F., Carson, J.S., Manivannan, M.S. (Eds.), Proceedings of the 1998 Winter Simulation Conference, pp. 701-706
[30] Kleijnen, J.P.C.; Van Ham, G.; Rotmans, J., Techniques for sensitivity analysis of simulation models: a case study of the CO2 greenhouse effect, Simulation, 58, 6, 410-417, (1992)
[31] Linhart, H., Zucchini, W., 1986. Model selection. Wiley, New York · Zbl 0665.62003
[32] Pierreval, H., 1996. A metamodel approach based on neural networks. International Journal in Computer Simulation 6 (3) 365-378
[33] Rao, C.R., Some problems involving linear hypothesis in multivariate analysis, Biometrika, 46, 49-58, (1959) · Zbl 0108.15405
[34] Sacks, J.; Welch, W.J.; Mitchell, T.J.; Wynn, H.P., Design and analysis of computer experiments (includes comments and rejoinder), Statistical science, 4, 4, 409-435, (1989) · Zbl 0955.62619
[35] Saltelli, A.; Andres, T.H.; Homma, T., Sensitivity analysis of model output; performance of the iterated fractional factorial design method, Computational statistics and data analysis, 20, 387-407, (1995) · Zbl 0900.62420
[36] Sanchez, S.M., Sanchez, P.J., Ramberg, J.S., Moeeni, F., 1996. Effective engineering design through simulation. International Transactions Operational Research 3 (2) 169-185
[37] Sargent, R.G., 1991. Research issues in metamodeling. In: Nelson, B.L., Kelton, W.D., Clark, G.M. (Eds.), Proceedings of the 1991 Winter Simulation Conference, pp. 888-893
[38] Sargent, R.G., 1996. Verifying and validating simulation models. In: Charnes, J.M., Morrice, D.M., Brunner, D.T., Swain, J.J. (Eds.), Proceedings of the 1996 Winter Simulation Conference, pp. 55-64
[39] Schlesinger, S, Terminology for model credibility, Simulation, 32, 3, 103-104, (1979)
[40] Van Groenendaal, W.J.H.; Kleijnen, J.P.C., On the assessment of economical risk: factorial design versus Monte Carlo methods, Journal of reliability engineering and systems safety, 57, 1, 103-105, (1997)
[41] Verkooyen, W.J.H., 1996. Neural networks in economic modelling. CentER, Tilburg University, Tilburg
[42] Welch, W.J.; Buck, R.J.; Sacks, J.; Wynn, H.P., Screening, predicting, and computer experiments, Technometrics, 34, 1, 15-25, (1992)
[43] Yu, B.; Popplewell, K., Metamodel in manufacturing: a review, International journal of production research, 32, 4, 787-796, (1994) · Zbl 0901.90123
[44] Zeigler, B., 1976. Theory of Modelling and Simulation. Wiley/Interscience, New York · Zbl 0352.68122
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.