×

Model calibration as a testing strategy for system dynamics models. (English) Zbl 1043.90019

Summary: System dynamics models are becoming increasingly common in the analysis of policy and managerial issues. The usefulness of these models is predicated on their ability to link observable patterns of behavior to micro-level structure and decision-making processes. This paper posits that model calibration – the process of estimating the model parameters (structure) to obtain a match between observed and simulated structures and behaviors – is a stringent test of a hypothesis linking structure to behavior, and proposes a framework to use calibration as a form of model testing. It tackles the issue at three levels: theoretical, methodological, and technical. First, it explores the nature of model testing, and suggests that the modeling process be recast as an experimental approach to gain confidence in the hypothesis articulated in the model. At the methodological level, it proposes heuristics to guide the testing strategy, and to take advantage of the strengths of automated calibration algorithms. Finally, it presents a set of techniques to support the hypothesis testing process. The paper concludes with an example and a summary of the argument for the proposed approach.

MSC:

90B25 Reliability, availability, maintenance, inspection in operations research
62F10 Point estimation

Software:

DYNAMO
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Barlas, Y., Multiple tests for validations of system dynamics type of simulation models, European Journal of Operational Research, 42, 1, 59-87 (1989) · Zbl 0712.65128
[2] Barlas, Y., Model validation in system dynamics, (Wolstenholme, E.; Monaghan, C., Proceedings of the 1994 International System Dynamics Conference (1994), System Dynamics Society: System Dynamics Society Stirling, Scotland), 1-10
[3] Barlas, Y.; Carpenter, S., Philosophical roots of model validation: Two paradigms, System Dynamics Review, 6, 2, 148-166 (1990)
[4] Bell, J. A.; Bell, J. F., System dynamics and scientific method, (Randers, J., Elements of the System Dynamics Method (1980), Productivity Press: Productivity Press Cambridge, MA), 3-22
[5] Bell, J. A.; Senge, P. M., Methods for enhancing refutability in system dynamics modeling, TIMS Studies in the Management Sciences, 14, 1, 61-73 (1980)
[6] Bunge, M., Scientific Research (1967), Springer-Verlag: Springer-Verlag New York · Zbl 0153.31201
[7] Checkland, P. B., Systems Thinking, Systems Practice (1981), Wiley: Wiley New York
[8] Cook, T. D.; Campbell, D. T., Quasi-Experimentation: Design & Analysis Issues for Field Settings (1979), Houghton Mifflin: Houghton Mifflin Boston
[9] Cyert, R.; March, J., A Behavioral Theory of the Firm (1963), Prentice Hall: Prentice Hall Englewood Cliffs, NJ
[10] Eden, C., Part III. Mixing methods-Introduction, (Eden, C.; Radford, J., Tackling Strategic Problems (1990), Sage Publications: Sage Publications London), 90-91
[11] (Flew, A., A Dictionary of Philosophy (1984), St. Martin’s Press: St. Martin’s Press New York)
[12] Forrester, J. W., Industrial Dynamics (1961), MIT Press: MIT Press Cambridge, MA
[13] Forrester, J. W., The model versus a modeling process, System Dynamics Review, 1, 2, 133-134 (1971)
[14] Forrester, J. W., An alternative approach to economic policy: Macrobehavior from microstructure, (Kamrany, N. M.; Day, R. H., Economic Issues of the Eighties (1979), John Hopkins University Press: John Hopkins University Press Baltimore), 80-108
[15] Forrester, J. W., Policies, decisions, information sources for modeling, (Morecroft, J. D.W.; Sterman, J. D., Modeling for Learning Organizations (1994), Productivity Press: Productivity Press Cambridge, MA), 51-84
[16] Forrester, J. W.; Senge, P. M., Tests for building confidence in system dynamics models, TIMS Studies in the Management Sciences, 14, 209-228 (1980)
[17] Gass, S. I., Decision-aiding models: Validation, assessment and related issues for policy analysis, Operations Research, 31, 4, 603-631 (1983)
[18] Graham, A. K., Parameter estimation in system dynamics modeling, (Randers, J., Elements of the System Dynamics Method (1980), Productivity Press: Productivity Press Cambridge, MA), 143-161
[19] Graham, A. K., On positioning system dynamics as an applied science of strategy, (Davidsen, P. I., Proceedings of the 2002 International System Dynamics Conference (2002), System Dynamics Society: System Dynamics Society Palermo, Italy)
[20] Graham, A. K.; Choi, C. Y.; Mullen, T. W., Using fit-constrained Monte Carlo trials to quantify confidence in simulation model outcomes, (Proceedings of the 35th Hawaii Conference on Systems Science (2002), IEEE: IEEE Big Island, HI)
[21] Greene, W. H., Econometric Analysis (1997), Prentice Hall: Prentice Hall Upper Saddle River, NJ
[22] Homer, J. B., Partial-model testing as a validation tool for system dynamics, (Proceedings of the1983 International System Dynamics Conference (1983), System Dynamics Society: System Dynamics Society Chestnut Hill, MA), 919-932
[23] Homer, J. B., Why we iterate: Scientific modeling in theory and practice, System Dynamics Review, 12, 1, 1-19 (1996)
[24] Homer, J. B., Structure, data and compelling conclusions: Notes from the field, System Dynamics Review, 13, 4, 293-309 (1997)
[25] Kennedy, P., A Guide to Econometrics (1992), MIT Press: MIT Press Cambridge, MA
[26] Kleijnen, J. P.C., Sensitivity analysis and optimization of system dynamics models: Regression analysis and statistical design of experiments, System Dynamics Review, 11, 4, 275-288 (1995)
[27] Kleijnen, J. P.C.; Sargent, R. G., A methodology for the fitting and validation of metamodels in simulation, European Journal of Operational Research, 120, 1, 14-29 (2000) · Zbl 0985.65007
[28] Kuhn, T. S., The Structure of Scientific Revolutions (1970), University of Chicago Press: University of Chicago Press Chicago
[29] Lakatos, I., Falsification and the methodology of scientific research programmes, (Lakatos, I.; Musgrave, A., Criticism and the Growth of Knowledge (1974), Cambridge University Press: Cambridge University Press Cambridge), 91-196
[30] Lane, D. C., Social theory and system dynamics practice, European Journal of Operational Research, 113, 3, 501-527 (1999) · Zbl 0949.91032
[31] Lant, T., Aspiration level adaptation: An empirical exploration, Management Science, 38, 5, 623-644 (1992)
[32] Levinthal, D.; March, J., A model of adaptive organizational search, Journal of Economic Behavior and Organization, 2, 4, 307-333 (1981)
[33] Long, J. S., Regression Models for Categorical and Limited Dependent Variables (1997), SAGE: SAGE Thousand Oaks, CA · Zbl 0911.62055
[34] Lyneis, J. M.; Pugh, A. L., Automated vs. ‘hand’ calibration of system dynamics models: An experiment with a simple project model, (Richardson, G. P.; Sterman, J. D., Proceedings of the 1996 International System Dynamics Conference (1996), System Dynamics Society: System Dynamics Society Cambridge, MA), 317-320
[35] Martinez, I. J.; Richardson, G. P., Best practices in system dynamics modeling, (Hines, J. H.; Diker, V. G., Proceedings of the 2001 International System Dynamics Conference (2001), System Dynamics Society: System Dynamics Society Atlanta)
[36] Mass, N. J.; Senge, P. M., Alternative test for selecting model variables, (Randers, J., Elements of the System Dynamics Method (1980), Productivity Press: Productivity Press Cambridge, MA), 203-223
[37] Miller, J. H., Active nonlinear tests (ANTs) of complex simulation models, Management Science, 44, 6, 820-830 (1998) · Zbl 0988.90527
[38] Miser, H. J., A foundational concept of science appropriate for validation in operational research, European Journal of Operational Research, 66, 2, 204-234 (1993)
[39] Mitroff, I., The myth of objectivity or why science needs a new psychology of science?, Management Science, 18, 10, B613-B618 (1972)
[40] Morecroft, J. D.W., System dynamics: Portraying bounded rationality, Omega, 11, 2, 131-142 (1983)
[41] Morecroft, J. D.W., Rationality in the analysis of behavioral simulation models, Management Science, 31, 7, 900-916 (1985)
[42] Morecroft, J. D.W., System dynamics and microworlds for policymakers, European Journal of Operational Research, 59, 1, 9-27 (1988)
[43] Oliva, R., 1995. A Vensim module to calculate summary statistics for historical fit. D-4584, System Dynamics Group, Massachusetts Institute of Technology, Cambridge, MA. Available from <http://www.people.hbs.edu/roliva/research/sd/; Oliva, R., 1995. A Vensim module to calculate summary statistics for historical fit. D-4584, System Dynamics Group, Massachusetts Institute of Technology, Cambridge, MA. Available from <http://www.people.hbs.edu/roliva/research/sd/
[44] Oliva, R., Empirical validation of a dynamic hypothesis, (Richardson, G. P.; Sterman, J. D., Proceedings of the 1996 International System Dynamics Conference (1996), System Dynamics Society: System Dynamics Society Cambridge, MA), 405-408
[45] Oliva, R., Tradeoffs in responses to work pressure in the service industry, California Management Review, 41, 4, 26-43 (2001)
[46] Oliva, R.; Sterman, J. D., Cutting corners and working overtime: Quality erosion in the service industry, Management Science, 47, 7, 894-914 (2001)
[47] Oreskes, N.; Shrader-Frechette, K.; Belitz, K., Verification, validation, and confirmation of numerical models in the earth sciences, Science, 263, 641-646 (1994)
[48] Peterson, D. W., Statistical tools for system dynamics, (Randers, J., Elements of the System Dynamics Method (1980), Productivity Press: Productivity Press Cambridge, MA), 143-161
[49] Raiffa, H.; Schlaifer, R., Applied Statistical Decision Theory (1961), Division of Research, Harvard Business School: Division of Research, Harvard Business School Boston · Zbl 0181.21801
[50] Reichelt, K. S.; Lyneis, J. M.; Bespolka, C. G., Calibration statistics: Selecting a statistic and setting a standard, (Richardson, G. P.; Sterman, J. D., Proceedings of the 1996 International System Dynamics Conference (1996), System Dynamics Society: System Dynamics Society Cambridge, MA), 425-428
[51] Richardson, G. P.; Pugh, A. L., Introduction to System Dynamics Modeling with DYNAMO (1981), MIT Press: MIT Press Cambridge, MA
[52] Roy, B., Decision science or decision-aid science?, European Journal of Operational Research, 66, 2, 184-203 (1993)
[53] Schweppe, F. C., Uncertain Dynamic Systems (1973), Prentice Hall: Prentice Hall Englewood Cliffs, NJ
[54] Senge, P. M., Statistical estimation of feedback models, Simulation, 28, June, 177-184 (1977)
[55] Senge, P.M., 1978. The system dynamics investment function: A comparison of the neoclassical investment function. PhD Thesis, Sloan School of Management, MIT, Cambridge, MA; Senge, P.M., 1978. The system dynamics investment function: A comparison of the neoclassical investment function. PhD Thesis, Sloan School of Management, MIT, Cambridge, MA
[56] Senge, P. M., The Fifth Discipline: The Art and Practice of the Learning Organization (1990), Doubleday Currency: Doubleday Currency New York
[57] Senge, P. M.; Oliva, R., Developing a theory of service quality/service capacity interaction, (Zepeda, E.; Machuca, J. A.D., Proceedings of the 1993 International System Dynamics Conference (1993), System Dynamics Society: System Dynamics Society Cancun, Mexico), 476-485
[58] Senge, P. M.; Sterman, J. D., Systems thinking and organizational learning: Acting locally and thinking globally in the organization of the future, European Journal of Operational Research, 59, 1, 137-150 (1992)
[59] Smith, J. H., Modeling muddles: Validation beyond the numbers, European Journal of Operational Research, 66, 2, 235-249 (1993)
[60] Sterman, J. D., Appropriate summary statistics for evaluating the historical fit of system dynamics models, Dynamica, 10, Winter, 51-66 (1984)
[61] Sterman, J. D., Modeling managerial behavior: Misperceptions of feedback in a dynamic decision making experiment, Management Science, 35, 3, 321-339 (1989)
[62] Sterman, J. D., Learning in and about complex systems, System Dynamics Review, 10, 2-3, 291-330 (1994)
[63] Sterman, J. D., Business Dynamics: Systems Thinking and Modeling for a Complex World (2000), Irwin McGraw-Hill: Irwin McGraw-Hill Boston
[64] Theil, H., Applied Economic Forecasting (1966), North-Holland Publishing: North-Holland Publishing Amsterdam
[65] van Horn, R. L., Validation of simulation results, Management Science, 17, 5, 247-258 (1971)
[66] Wilson, J. R., Doctoral colloquium keynote address: Conduct, misconduct, cargo cult science, (Andradóttir, S.; Healy, K. J.; etal., Proceedings of the 1997 Winter Simulation Conference (1997), IEEE: IEEE Piscataway, NJ), 1405-1413
[67] Wolstenholme, E. F., System Enquiry: A System Dynamics Approach (1990), Wiley: Wiley New York
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.