zbMATH — the first resource for mathematics

Towards a new framework for evaluating systemic problem structuring methods. (English) Zbl 1317.90147
Summary: Operational researchers and social scientists often make significant claims for the value of systemic problem structuring and other participative methods. However, when they present evidence to support these claims, it is usually based on single case studies of intervention. There have been very few attempts at evaluating across methods and across interventions undertaken by different people. This is because, in any local intervention, contextual factors, the skills of the researcher and the purposes being pursued by stakeholders affect the perceived success or failure of a method. The use of standard criteria for comparing methods is therefore made problematic by the need to consider what is unique in each intervention. So, is it possible to develop a single evaluation approach that can support both locally meaningful evaluations and longer-term comparisons between methods? This paper outlines a methodological framework for the evaluation of systemic problem structuring methods that seeks to do just this.

90B50 Management decision making, including multiple objectives
Full Text: DOI
[1] Adams, R.; McCullough, A., The urban practitioner and participation in research within a streetwork context, Community, Work & Family, 6, 269-287, (2003)
[2] Alberts, D. J., Stakeholders or subject matter experts, who should be consulted?, Energy Policy, 35, 2336-2346, (2007)
[3] Allsop, J.; Taket, A., Evaluating user involvement in primary healthcare, International Journal of Healthcare Technology & Management, 5, 34-44, (2003)
[4] Alrøe, H. F., Science as systems learning: some reflections on the cognitive and communicational aspects of science, Cybernetics and Human Knowing, 7, 57-78, (2000)
[5] Baker, V., Gregory, W., Midgley, G., Veth, J., 2006. Ethical Implications and Social Impacts of Forensic DNA Technologies and Applications: Summary Report. Institute of Environmental Science and Research (ESR) Ltd., Christchurch.
[6] Baker, V., Midgley, G., 2007. Review of the MoRST Roadmaps Exercise: Final Report. Confidential ESR Client Report. Institute of Environmental Science and Research (ESR) Ltd., Wellington.
[7] Bateson, G., Form, substance, and difference, General Semantics Bulletin, 37, 5-13, (1970)
[8] Beierle, T. C.; Cayford, J., Democracy in practice: public participation in environmental decisions, (2002), RFF Press Washington, DC
[9] Beierle, T. C.; Konisky, D. M., Values, conflict, and trust in participatory environmental planning, Journal of Policy Analysis and Management, 19, 587-602, (2000)
[10] Berry, H.; Bowman, S. R.; Hernandez, R.; Pratt, C., Evaluation tool for community development coalitions, Journal of Extension, 44, (2006), http://www.joe.org/joe/2006december/tt2.shtml (accessed: 30.03.07)
[11] Bjärås, G.; Haglund, B. J.A.; Rifkin, S. B., A new approach to community participation assessment, Health Promotion International, 6, 199-206, (1991)
[12] Branch, K. M.; Bradbury, J. A., Comparison of DOE and army advisory boards: application of a conceptual framework for evaluating public participation in environmental risk decision making, Policy Studies Journal, 34, 723-753, (2006)
[13] Brocklesby, J., Becoming multimethodology literate: an assessment of the cognitive difficulties of working across paradigms, (Mingers, J.; Gill, A., Multimethodology: The Theory and Practice of Combining Management Science Methodologies, (1997), Wiley Chichester)
[14] Brocklesby, J., Ethics beyond the model: how social dynamics can interfere with ethical practice in operational research / management science, Omega, the International Journal of Management Science, 37, 1073-1082, (2009)
[15] Bryant, J. W.; Darwin, J. A., Exploring inter-organisational relationships in the health service: an immersive drama approach, European Journal of Operational Research, 152, 655-666, (2004) · Zbl 1043.90522
[16] Burns, D., Systemic action research: A strategy for whole system change, (2007), Policy Press Bristol
[17] Buysse, V.; Wesley, P.; Skinner, D., Community development approaches for early intervention, Topics in Early Childhood Special Education, 19, 236-243, (1999)
[18] Cavana, R. Y.; Delahaye, B. L.; Sekaran, U., Applied business research: qualitative and quantitative methods, (2001), Wiley Brisbane
[19] Champion, D.; Wilson, J. M., The impact of contingency factors on validation of problem structuring methods, Journal of the Operational Research Society, 61, 1420-1431, (2010)
[20] Charnley, S.; Engelbert, B., Evaluating public participation in environmental decision-making: EPA’s superfund community involvement program, Journal of Environmental Management, 77, 165-182, (2005)
[21] Checkland, P., Systems Thinking, Systems Practice, (1981), Wiley Chichester
[22] Checkland, P. B.; Forbes, P.; Martin, S., Techniques in soft systems practice. part 3: monitoring and control in conceptual models and in evaluation studies, Journal of Applied Systems Analysis, 17, 29-37, (1990)
[23] Checkland, P.; Scholes, J., Soft systems methodology in action, (1990), Wiley Chichester
[24] Checkland, P.; Poulter, J., Learning for action, (2006), Wiley Chichester
[25] Chess, C.; Purcell, K., Public participation and the environment: do we know what works?, Environmental Science & Technology, 33, 2685-2692, (1999)
[26] Churchman, C. W., Operations research as a profession, Management Science, 17, B37-53, (1970)
[27] Clayton, A. M.H.; Radcliffe, N. J., Sustainability: A systems approach, (1996), Earthscan London
[28] Cole, M., Evaluating the impact of community appraisals: some lessons from south-west england, Policy & Politics, 34, 51-68, (2006)
[29] Connell, N. A.D., Evaluating soft OR: some reflections on an apparently ‘unsuccessful’ implementation using a soft systems methodology (SSM) based approach, Journal of the Operational Research Society, 52, 150-160, (2001) · Zbl 1131.90301
[30] Delaney, M. M.; Foroughi, A.; Perkins, W. C., An empirical study of the efficacy of a computerized negotiation support system (NSS), Decision Support Systems, 20, 185-197, (1997)
[31] De Vreede, G.; Dickson, G., Using GSS to design organizational processes and information systems: an action research study on collaborative business engineering, Group Decision and Negotiation, 9, 161-183, (2000)
[32] Douglas, M., How institutions think, (1986), Routledge and Kegan Paul London
[33] Duignan, P.; Casswell, S., Evaluating community development programs for health promotion: problems illustrated by a New Zealand example, Community Health Studies, 13, 74-81, (1989)
[34] Duram, L. A.; Brown, K. G., Assessing public participation in U.S. watershed planning initiatives, Society & Natural Resources, 12, 455-467, (1999)
[35] Eden, C., On evaluating the performance of ‘wide-band’ GDSS, European Journal of Operational Research, 81, 302-311, (1995) · Zbl 0927.90059
[36] Eden, C.; Ackermann, F., “horses for courses”: a stakeholder approach to the evaluation of gdsss, Group Decision and Negotiation, 5, 501-519, (1996)
[37] Eden, C.; Ackermann, F., Where next for problem structuring methods, Journal of the Operational Research Society, 57, 766-768, (2006)
[38] Eden, C.; Sims, D., On the nature of problems in consulting practice, Omega, 7, 119-127, (1979)
[39] Entwistle, V.; Buchan, H.; Coulter, A.; Jadad, A., Towards constructive innovation and rigorous evaluation: a new series on methods for promoting and evaluating participation, Health Expectations, 2, 75-77, (1999)
[40] Er, M. C.; Ng, A. C., The anonymity and proximity factors in group decision support systems, Decision Support Systems, 14, 75-83, (1995)
[41] Fan, S.; Shen, Q.; Lin, G., Comparative study of idea generation between traditional value management workshops and GDSS-supported workshops, Journal of Construction Engineering and Management, 133, 816-825, (2007)
[42] Fjermestad, J., An analysis of communication mode in group support systems research, Decision Support Systems, 37, 239-263, (2004)
[43] Fjermestad, J.; Hiltz, S., An assessment of group support systems experimental research: methodology and results, Journal of Management Information Systems, 15, 7-149, (1998)
[44] Flood, R. L., Solving problem solving, (1995), Wiley Chichester
[45] (Flood, R. L.; Jackson, M. C., Critical Systems Thinking: Directed Readings, (1991), Wiley Chichester)
[46] (Flood, R. L.; Romm, N. R.A., Critical Systems Thinking: Current Research and Practice, (1996), Plenum New York)
[47] Forrester, J. W., Principles of systems, (1969), Wright-Allen Press Cambridge, MA
[48] Franco, L. A., Forms of conversation and problem structuring methods: a conceptual development, Journal of the Operational Research Society, 57, 813-821, (2006) · Zbl 1103.90373
[49] Franco, L. A., Assessing the impact of problem structuring methods in multi-organizational settings: an empirical investigation, Journal of the Operational Research Society, 58, 760-768, (2007) · Zbl 1175.90270
[50] Franco, L. A.; Shaw, D.; Westcombe, M., Taking problem structuring methods forward, Journal of the Operational Research Society, 58, 545-546, (2007)
[51] Gallupe, R. B.; Dennis, A. R.; Cooper, W. H.; Valacich, J. S.; Bastianutti, L. M.; Nunamaker, J. F., Electronic brainstorming and group size, Academy of Management Journal, 35, 350-369, (1992)
[52] Gopal, A.; Prasad, P., Understanding GDSS in symbolic context: shifting the focus from technology to interaction, MIS Quarterly, 24, 509-546, (2000)
[53] Halvorsen, K. E., Assessing public participation techniques for comfort, convenience, satisfaction, and deliberation, Environmental Management, 28, 179-186, (2001)
[54] Hepi, M.; Foote, J.; Ahuriri-Driscoll, A., Guidelines for developing resource care evaluation criteria and methods, (2008), Institute of Environmental Science and Research (ESR) Ltd. Christchurch
[55] Hjortsø, C. N., Enhancing public participation in natural resource management using soft OR: an application of strategic option development and analysis in tactical forest planning, European Journal of Operational Research, 152, 667-683, (2004) · Zbl 1043.90042
[56] Ho, C.H., 1997. A Critical Process for the Evaluation of Methodology. Ph.D. Thesis, University of Hull.
[57] Jackson, M. C., Systems methodology for the management sciences, (1991), Plenum New York
[58] Jackson, M. C., Systems approaches to management, (2000), Plenum/Kluwer New York
[59] Jackson, M. C., Beyond problem structuring methods: reinventing the future of OR/MS, Journal of the Operational Research Society, 57, 868-878, (2006) · Zbl 1103.90375
[60] Jackson, M. C.; Keys, P., Towards a system of systems methodologies, Journal of the Operational Research Society, 35, 473-486, (1984)
[61] Jenkins, N. T.; Bennett, M. I.J., Toward an empowerment zone evaluation, Economic Development Quarterly, 13, 23-28, (1999)
[62] Joldersma, C.; Roelofs, E., The impact of soft OR-methods on problem structuring, European Journal of Operational Research, 152, 696-708, (2004) · Zbl 1043.90514
[63] Kelly, K.; Van Vlaenderen, H., Evaluating participation processes in community development, Evaluation & Program Planning, 18, 371-383, (1995)
[64] Keys, P., Understanding the process of operational research: collected readings, (1994), Wiley Chichester
[65] Li, X.; Zheng, H., Study on general systems methodology, (Midgley, G.; Wilby, J., Systems Methodology: Possibilities for Cross-Cultural Learning and Integration, (1995), Centre for Systems Studies Hull)
[66] Luhmann, N., Ecological communication, (1986), University of Chicago Press Chicago
[67] Maani, K. E.; Cavana, R. Y., Systems thinking, system dynamics, (2007), Pearson New Zealand Auckland
[68] Margerum, R. D., Collaborative planning: building consensus and building a distinct model for practice, Journal of Planning Education & Research, 21, 237-253, (2002)
[69] Masozera, M. K.; Alavalapati, J. R.R.; Jacobson, S. K.; Shrestha, R. K., Assessing the suitability of community-based management for the nyungwe forest reserve, rwanda, Forest Policy & Economics, 8, 206-216, (2006)
[70] McAllister, K., 1999. Understanding Participation: Monitoring and Evaluating Process, Outputs and Outcomes. Working Paper 2, IDRC, Ottawa.
[71] McCartt, A. T.; Rohrbaugh, J., Managerial openness to change and the introduction of GDSS: explaining initial success and failure in decision conferencing, Organization Science, 6, 569-584, (1995)
[72] McGurk, B.; Sinclair, A. J.; Diduck, A., An assessment of stakeholder advisory committees in forest management: case studies from manitoba, Canada, Society & Natural Resources, 19, 809-826, (2006)
[73] McKay, J., Using cognitive mapping to achieve shared understanding in information requirements determination, Australian Computer Journal, 30, 139-145, (1998)
[74] Midgley, G., Ecology and the poverty of humanism: a critical systems perspective, Systems Research, 11, 67-76, (1994)
[75] Midgley, G., Systemic intervention: philosophy, methodology, and practice, (2000), Plenum/Kluwer New York
[76] (Midgley, G., Systems Thinking, vols. I to IV, (2003), Sage London)
[77] Midgley, G., Theoretical pluralism in systemic action research, Systemic Practice and Action Research, 24, 1-15, (2011)
[78] Midgley, G.; Ahuriri-Driscoll, A.; Baker, V.; Foote, J.; Hepi, M.; Taimona, H.; Rogers-Koroheke, M.; Gregor, J.; Gregory, W.; Lange, M.; Veth, J.; Winstanley, A.; Wood, D., Practitioner identity in systemic intervention: reflections on the promotion of environmental health through Māori community development, Systems Research and Behavioral Science, 24, 233-247, (2007)
[79] Midgley, G., Winstanley, A., Gregory, W., Foote, J., 2005. Scoping the Potential Uses of Systems Thinking in Developing Policy on Illicit Drugs. Drug Policy Modelling Project Research Memorandum #13. Turning Point, Melbourne.
[80] Mingers, J. C., Towards critical pluralism, (Mingers, J.; Gill, A., Multimethodology: The Theory and Practice of Combining Management Science Methodologies, (1997), Wiley Chichester)
[81] Mingers, J.; Brocklesby, J., Multimethodology: towards a framework for mixing methodologies, Omega, the International Journal of Management Science, 25, 489-509, (1997)
[82] (Mingers, J.; Gill, A., Multimethodology: The Theory and Practice of Combining Management Science Methodologies, (1997), Wiley Chichester)
[83] Mingers, J.; Rosenhead, J., Problem structuring methods in action, European Journal of Operational Research, 152, 530-554, (2004) · Zbl 1044.90027
[84] Montazemi, A. R.; Wang, F.; Nainar, S. M.K.; Bart, C. K., On the effectiveness of decisional guidance, Decision Support Systems, 18, 181-198, (1996)
[85] Morgan, L. M., Community participation in health: perpetual allure, persistent challenge, Health Policy & Planning, 16, 221-230, (2001)
[86] Murphy-Berman, V.; Schnoes, C.; Chambers, J. M., An early stage evaluation model for assessing the effectiveness of comprehensive community initiatives: three case studies in nebraska, Evaluation & Program Planning, 23, 157-163, (2000)
[87] Nunamaker, J. F.; Dennis, A. R.; Valacich, J. S.; Vogel, D. R.; George, J. F., Electronic meeting systems to support group work, Communications of the ACM, 34, 43-61, (1991)
[88] Ong, B. N., Assessing the context for partnerships between communities and the national health service in england, Critical Public Health, 10, 343-351, (2000)
[89] Paterson, J.; Teubner, G., Changing maps: empirical legal autopoiesis, Social and Legal Studies, 7, 451-486, (1998)
[90] Pettigrew, A. M., Context and action in the transformation of the firm, Journal of Management Studies, 24, 649-670, (1987)
[91] Phahlamohlaka, J.; Friend, J., Community planning for rural education in south africa, European Journal of Operational Research, 152, 684-695, (2004) · Zbl 1043.90517
[92] Pinsonneault, A.; Barki, H.; Gallupe, R. B.; Hoppen, M., Electronic brainstorm: the illusion of productivity, Information Systems Research, 10, 110-132, (1999)
[93] Pinsonneault, A.; Kraemer, K. L., The effects of electronic meetings on group processes and outcomes: an assessment of the empirical research, European Journal of Operational Research, 46, 143-161, (1990) · Zbl 06908647
[94] Romm, N. R.A., Inquiry-and-intervention in systems planning: probing methodological rationalities, World Futures, 47, 25-36, (1996)
[95] Rosenhead, J., Rational analysis for a problematic world, (1989), Wiley Chichester
[96] Rosenhead, J., Past, present and future of problem structuring methods, Journal of the Operational Research Society, 57, 759-765, (2006) · Zbl 1103.90301
[97] Rosenhead, J.; Mingers, J., Rational analysis for a problematic world revisited, (2001), Wiley Chichester
[98] Rosenhead, J.; Mingers, J., Problem structuring methods in action, European Journal of Operational Research, 152, 530-554, (2004) · Zbl 1044.90027
[99] Rouwette, E. A.J. A., Facilitated modelling in strategy development: measuring the impact on communication, consensus and commitment, Journal of the Operational Research Society, 62, 879-887, (2011)
[100] Rouwette, E.; Bastings, I.; Blokker, H., A comparison of group model building and strategic options development and analysis, Group Decision and Negotiation, 20, 781-803, (2011)
[101] Rouwette, E. A.J. A.; Vennix, J. A.M.; Felling, A. J.A., On evaluating the performance of problem structuring methods: an attempt at formulating a conceptual model, Group Decision and Negotiation, 18, 567-587, (2009)
[102] Rowe, G.; Frewer, L. J., Public participation methods: a framework for evaluation, Science, Technology & Human Values, 25, 3-29, (2000)
[103] Rowe, G.; Frewer, L. J., Evaluating public participation exercises: a research agenda, Science, Technology & Human Values, 29, 512-556, (2004)
[104] Rowe, G.; Horlick-Jones, T.; Walls, J.; Pidgeon, N., Difficulties in evaluating public engagement initiatives: reflections on an evaluation of the UK GM nation? public debate about transgenic crops, Public Understanding of Science, 14, 331-352, (2005)
[105] Rowe, G.; Marsh, R.; Frewer, L. J., Evaluation of a deliberative conference, Science, Technology & Human Values, 29, 88-121, (2004)
[106] Shaw, D., Evaluating electronic workshops through analysing the ‘brainstormed’ ideas, Journal of the Operational Research Society, 54, 692-705, (2003) · Zbl 1060.90674
[107] Shaw, D.; Franco, A.; Westcombe, M., Problem structuring methods: new directions in a problematic world, Journal of the Operational Research Society, 57, 757-758, (2006)
[108] Shaw, I., Qualitative evaluation, (1999), Sage London
[109] Shen, C-Y.; Midgley, G., Toward a buddhist systems methodology 1: comparisons between buddhism and systems theory, Systemic Practice and Action Research, 20, 167-194, (2007)
[110] Sieber, R., Public participation geographic information systems: a literature review and framework, Annals of the Association of American Geographers, 96, 491-507, (2006)
[111] (Smith, L. T., Decolonizing Methodologies: Research and Indigenous Peoples, (1999), Zed Books London)
[112] Sørensen, L.; Vidal, R.; Engström, E., Using soft OR in a small company – the case of kirby, European Journal of Operational Research, 152, 555-570, (2004) · Zbl 1043.90532
[113] Spash, C. L., Ethics and environmental attitudes with implications for economic valuation, Journal of Environmental Management, 50, 403-416, (1997)
[114] Sykes, C.; Goodwin, W., Assessing patient, carer and public involvement in health care, Quality in Primary Care, 15, 45-52, (2007)
[115] Taket, A.; White, L., Partnership and participation: decision-making in the multi-agency setting, (2000), Wiley Chichester · Zbl 1055.90580
[116] Tuler, S.; Webler, T.; Finson, R., Competing perspectives on public involvement: planning for risk characterization and risk communication about radiological contamination from a national laboratory, Health, Risk & Society, 7, 247-266, (2005)
[117] Ulrich, W., Critical heuristics of social planning: A new approach to practical philosophy, (1994), Wiley Chichester
[118] Valacich, J. S.; Schwenk, C., Devil’s advocacy and dialectical inquiry effects on face-to-face and computer-mediated group decision making, Organizational Behavior and Human Decision Processes, 63, 158-173, (1995)
[119] Valacich, J. S.; Schwenk, C., Structuring conflict in individual, face-to-face, and computer-mediated group decision making: carping versus objective devil’s advocacy, Decision Sciences, 26, 369-393, (1995)
[120] Vennix, J. A.M., Group model building, (1996), Wiley Chichester
[121] Warburton, D.; Wilson, R.; Rainbow, E., Making a difference: A guide to evaluating public participation in central government, (2007), Involve London, http://www.involve.org.uk/evaluation (accessed: 30.05.07)
[122] White, L., Evaluating problem-structuring methods: developing an approach to show the value and effectiveness of psms, Journal of the Operational Research Society, 57, 842-855, (2006) · Zbl 1103.90381
[123] Winstanley, A.; Baker, V.; Foote, J.; Gregor, J.; Gregory, W.; Hepi, M.; Midgley, G., Water in the waimea basin: community values and water management options, (2005), Institute of Environmental Science and Research (ESR) Ltd. Christchurch
[124] Yearley, S., Bridging the science-policy divide in urban air-quality management: evaluating ways to make models more robust through public engagement, Environment and Planning C, 24, 701-714, (2006)
[125] Zhang, J.; Smith, R.; Watson, R. B., Towards computer support of the soft systems methodology: an evaluation of the functionality and usability of an SSM toolkit, European Journal of Information Systems, 6, 129-139, (1997)
[126] Zhu, Z., Dealing with a differentiated whole: the philosophy of the WSR approach, Systemic Practice and Action Research, 13, 21-57, (2000)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.