×

Aggregating disparate estimates of chance. (English) Zbl 1127.62129

Summary: We consider a panel of experts asked to assign probabilities to events, both logically simple and complex. The events evaluated by different experts are based on overlapping sets of variables but may otherwise be distinct. The union of all the judgments will likely be probabilistically incoherent. We address the problem of revising the probability estimates of the panel so as to produce a coherent set that best represents the group’s expertise.

MSC:

62P99 Applications of statistics
62-07 Data analysis (statistics) (MSC2010)
90B50 Management decision making, including multiple objectives
62C99 Statistical decision theory

Software:

Genocop
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Alpert, M.; Raiffa, H., A progress report on the training of probability assessors, (Kahneman, D.; Slovic, P.; Tversky, A., Judgment Under Uncertainty: Heuristics and Biases (1982), Cambridge Univ. Press: Cambridge Univ. Press New York, NY), 294-305
[2] Ariely, D.; Au, W. T.; Bender, R. H.; Budescu, D. V.; Dietz, C. B.; Gu, H.; Wallsten, T. S.; Zauberman, G., The Effects of Averaging Subjective Probability Estimates Between and Within Judges, J. Exper. Psychol.: Appl., 6, 2, 130-147 (2000)
[3] Ashton, A. H.; Ashton, R. H., Aggregating subjective forecasts: some empirical results, Manage. Sci., 31, 1499-1508 (1985)
[4] Bahar, R.; Frohm, E.; Gaona, C.; Hachtel, G.; Macii, E.; Pardo, A.; Somenzi, F., Algebraic decision diagrams and their applications, J. Formal Methods Systems Design, 10, 2/3, 171-206 (1997)
[5] Bar-Hillel, M., On the subjective probability of compound events, Organ. Behav. Human Performance, 9, 396-406 (1973)
[6] Baron, J., Thinking and Deciding (2000), Cambridge Univ. Press: Cambridge Univ. Press Cambridge, UK
[7] Batsell, R.; Brenner, L.; Osherson, D.; Tsavachidis, S.; Vardi, M. Y., Eliminating incoherence from subjective estimates of chance, (Proceedings of the 8th International Conference on the Principles of Knowledge Representation and Reasoning. Proceedings of the 8th International Conference on the Principles of Knowledge Representation and Reasoning, (KR 2002) (2002)), 353-364
[8] Berend, D.; Paroush, J., When is Condorcet’s jury theorem valid?, Soc. Choice Welfare, 15, 481-488 (1998) · Zbl 1066.91528
[9] Bernardo, J.; Smith, A., Bayesian Theory (1994), Wiley: Wiley New York · Zbl 0796.62002
[10] Biazzo, V.; Gilio, G., A generalization of the fundamental theorem of de Finetti for imprecise conditional probability assessments, Int. J. Approx. Reasoning, 24, 251-272 (2000) · Zbl 0995.68124
[11] Biazzo, V., Gilio, A., Lukasiewicz, T., Sanfilippo, G., 2001. Probabilistic logic under coherence: Complexity and algorithms. In: 2nd International Symposium on Imprecise Probabilities and Their Applications. Ithaca, NY; Biazzo, V., Gilio, A., Lukasiewicz, T., Sanfilippo, G., 2001. Probabilistic logic under coherence: Complexity and algorithms. In: 2nd International Symposium on Imprecise Probabilities and Their Applications. Ithaca, NY · Zbl 1001.68146
[12] Bonini, N.; Tentori, K.; Osherson, D., A New Conjunction Fallacy (2004), Mind & Language
[13] Brier, G., Verification of forecasts expressed in terms of probability, Monthly Weather Rev., 78, 1-3 (1950)
[14] Camerer, C. F., General conditions for the success of bootstrapping models, Organ. Behav. Human Performance, 27, 411-422 (1981)
[15] Castillo, E.; Gutiérrez, J.; Hadi, A., Expert Systems and Probabilistic Network Models (1997), Springer: Springer New York
[16] Chvátal, V., Linear Programming (1983), W.H. Freeman: W.H. Freeman San Francisco, CA · Zbl 0318.05002
[17] Clemen, R. T., Combining forecasts: A review and annotated bibliography, Int. J. Forecasting, 5, 559-583 (1989)
[18] Clemen, R. T.; Winkler, R. L., Aggregating point estimates: A flexible modeling approach, Manage. Sci., 39, 4, 501-515 (1993) · Zbl 0775.90247
[19] Clemen, R. T.; Winkler, R. L., Combining probability distributions from experts in risk analysis, Risk Analysis, 19, 187-203 (1999)
[20] Clemen, R. T.; Jones, S. K.; Winkler, R. L., Aggregating forecasts: An empirical evaluation of some Bayesian methods, (Berry, K. M.C.; Geweke, J. K., Bayesian Analysis in Statistics and Economics (1996), Wiley: Wiley New York, NY)
[21] Coletti, G.; Gilio, A.; Scozzafava, R., Comparative probability for conditional events: A new look through coherence, Theory Dec., 35, 237-258 (1993) · Zbl 0785.90006
[22] Cooke, R., Experts in Uncertainty (1991), Oxford Univ. Press: Oxford Univ. Press New York
[23] Cover, T.; Thomas, J., Elements of Information Theory (1991), Wiley: Wiley New York, NY · Zbl 0762.94001
[24] Dawes, R. M., The robust beauty of improper linear models, Amer. Psychol., 34, 571-582 (1979)
[25] Dawes, R. M.; Corrigan, B., Linear models in decision making, Psychol. Bull., 81, 97-106 (1974)
[26] de Finetti, B., Theory of Probability, vol. 1 (1974), Wiley: Wiley New York, NY · Zbl 0328.60002
[27] Deines, J., Osherson, D., Thompson, J., Tsavachidis, S., Vardi, M.Y., 2003. Removing incoherence from subjective estimates of chance. Technical report. Rice University. Available at http://www.princeton.edu/ osherson/Papers/cohere.pdf; Deines, J., Osherson, D., Thompson, J., Tsavachidis, S., Vardi, M.Y., 2003. Removing incoherence from subjective estimates of chance. Technical report. Rice University. Available at http://www.princeton.edu/ osherson/Papers/cohere.pdf
[28] Dietrich, F., List, C., April 2004. Strategy-proof judgment aggregation. Manuscript; Dietrich, F., List, C., April 2004. Strategy-proof judgment aggregation. Manuscript
[29] Druzdzel, M. J.; van der Gaag, L. C., Elicitation of probabilities for belief networks: Combining qualitative and quantitative information, (Uncertainty in Artificial Intelligence (95): Proceedings of the 11th conference (1995), Morgan Kaufmann: Morgan Kaufmann Los Altos, CA), 112-139
[30] Fagin, R.; Halpern, J.; Megiddo, N., A Logic for Reasoning about Probabilities, Info. Comput., 87, 78-128 (1990) · Zbl 0811.03014
[31] Ferrell, W. R., Combining individual judgments, (Wright, G., Behavioral Decision Making (1985), Plenum Press: Plenum Press New York, NY), 111-145
[32] Fletcher, R., Practical Methods of Optimization (1986), Wiley · Zbl 0905.65002
[33] Fong, G. T.; Krantz, D. H.; Nisbett, R. E., The effects of statistical training about everyday problems, Cogn. Psychol., 18, 253-292 (1986)
[34] Gärdenfors, P., Knowledge in Flux: Modeling the Dynamics of Epistemic States (1988), MIT Press: MIT Press Cambridge, MA · Zbl 1229.03008
[35] Genest, C.; Zidek, J., Combining probability distributions: A critique and an annotated bibliography, Statist. Sci., 1, 1, 114-135 (1986) · Zbl 0587.62017
[36] Georgakopoulos, G.; Kavvadias, D.; Papadimitriou, C., Probabilistic satisfiability, J. Complexity, 4, 1-11 (1988) · Zbl 0647.68049
[37] (Gilovich, T.; Griffin, D.; Kahneman, D., Heuristics and Biases: The Psychology of Intuitive Judgment (2002), Cambridge Univ. Press: Cambridge Univ. Press New York, NY)
[38] Grofman, B. G.; Owen, S. L.; Feld, S. L., Thirteen theorems in search of the truth, Theory Dec., 15, 261-278 (1983) · Zbl 0511.90011
[39] Gustason, W., Reasoning from Evidence: Inductive Logic (1994), Macmillan Co.: Macmillan Co. New York, NY
[40] Hailperin, T., Sentential Probability Logic (1996), Lehigh Univ. Press: Lehigh Univ. Press Bethlehem, PA · Zbl 0922.03026
[41] Halpern, J. Y., Reasoning about Uncertainty (2003), MIT Press: MIT Press Cambridge, MA · Zbl 1090.68105
[42] Hansson, S. O., A Textbook of Belief Dynamics: Theory Change and Database Updating (1999), Kluwer Academic: Kluwer Academic Dordrecht · Zbl 0947.03023
[43] Hastie, R.; Dawes, R. M., Rational Choice in an Uncertain World: The Psychology of Judgment and Decision Making (2001), Sage: Sage Thousand Oaks, CA
[44] Henrion, M., 1987. Practical issues in constructing a Bayes’ belief network. Third Workshop in Uncertainty and Artificial Intelligence, pp. 132-139; Henrion, M., 1987. Practical issues in constructing a Bayes’ belief network. Third Workshop in Uncertainty and Artificial Intelligence, pp. 132-139
[45] Hogarth, R., A note on aggregating opinions, Organ. Behav. Human Performance, 21, 42-43 (1978)
[46] Holtzman, S.; Breese, J., Exact reasoning about uncertainty: On the design of expert systems for decision support, (Lemmer, J. F.; Kanal, L. N., Uncertainty and Artificial Intelligence (1986), North-Holland/Elsevier: North-Holland/Elsevier New York, NY), 339-346
[47] Homer, S.; Selman, A. L., Computability and Complexity Theory (2001), Springer: Springer New York, NY · Zbl 1033.68045
[48] Howard, R., Knowledge maps, Manage. Sci., 35, 903-922 (1989)
[49] Jaumard, B.; Hansen, P.; de Aragao, M. P., Column generation methods for probabilistic logic, ORSA J. Comput., 3, 2, 135-148 (1991) · Zbl 0800.68864
[50] Jeffrey, R. C., The Logic of Decision (1983), Univ. of Chicago Press: Univ. of Chicago Press Chicago, IL
[51] Johnson, P. E., Social Choice: Theory and Research (1998), Sage: Sage New York, NY · Zbl 0984.91034
[52] Johnson, T.; Budescu, D.; Wallsten, T., Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value, J. Behav. Dec. Making, 14, 123-140 (2001)
[53] Joyce, J. M., A Nonpragmatic Vindication of Probabilism, Philosophy Sci., 65, 575-603 (1998)
[54] (Kahneman, D.; Tversky, A., Choices, Values, and Frames (2000), Cambridge Univ. Press: Cambridge Univ. Press New York, NY) · Zbl 1225.91017
[55] Kelly, J. S., Arrow Impossibility Theorems (1978), Academic Press: Academic Press New York, NY · Zbl 0462.90004
[56] Kirkpatrick, S.; Gelatt, C. D.; Vecchi, M. P., Optimization by simulated annealing, Science, 220, 671-680 (1983) · Zbl 1225.90162
[57] Klayman, J.; Brown, K., Debias the environment instead of the judge: An alternative approach to reducing error in diagnostic (and other) judgment, Cognition, 49, 97-122 (1993)
[58] Kornhauser, L. A.; Sager, L. G., Unpacking the court, Yale Law J., 96, 1, 82-117 (1986)
[59] Lad, F., Operational Subjective Statistical Methods (1996), Wiley: Wiley New York, NY · Zbl 0862.62005
[60] Ladha, K. K., Information polling through majority rule voting: Condorcet’s jury theorem with correlated votes, J. Econ. Behav. Organ., 26, 353-372 (1995)
[61] Lindley, D. V.; Tversky, A.; Brown, R. V., On the reconciliation of probability assessments, J. Royal Statist. Soc. A, 142, Part 2, 146-180 (1979) · Zbl 0427.62003
[62] List, C., A possibility theorem on aggregation over multiple interconnected propositions, Math. Soc. Sci., 45, 1, 1-13 (2003) · Zbl 1031.91025
[63] List, C., The probability of inconsistencies in complex collective decisions, Soc. Choice Welfare, 24, 1, 3-32 (2005) · Zbl 1100.91020
[64] List, C.; Pettit, P., Aggregating sets of judgments: An impossibility result, Econ. Philosophy, 18 (2002)
[65] Lord, C. G.; Ross, L.; Lepper, M., Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence, J. Personality Soc. Psychol., 37, 2098-2109 (1979)
[66] Luenberger, D. G., Linear and Nonlinear Programming (1984), Addison-Wesley: Addison-Wesley Reading, MA · Zbl 0241.90052
[67] Michalewicz, Z., Genetic Algorithms + Data Structures = Evolution Programs (1992), Springer-Verlag: Springer-Verlag Berlin · Zbl 0763.68054
[68] Mitchell, M., An Introduction to Genetic Algorithms (1996), MIT Press: MIT Press Cambridge, MA
[69] Morgan, M. G.; Henrion, M., Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis (1990), Cambridge Univ. Press: Cambridge Univ. Press Cambridge, England
[70] Morris, P., Decision analysis expert use, Manage. Sci., 20, 1233-1241 (1974) · Zbl 0317.90002
[71] Nilsson, N., Probabilistic logic, Artificial Intelligence, 28, 1, 71-87 (1986) · Zbl 0589.03007
[72] Nisbett, R. E.; Fong, G. T.; Lehman, D. R.; Cheng, P. W., Teaching reasoning, Science, 238, 625-631 (1987)
[73] Osherson, D., Probability judgment, (Smith, E. E.; Osherson, D., Thinking (1995), MIT Press: MIT Press Cambridge, MA), 35-76
[74] Parenté, F. J.; Anderson-Parenté, J. K., Delphi inquiry systems, (Wright, G.; Ayton, P., Judgmental Forecasting (1987), Wiley: Wiley New York, NY), 129-156
[75] Pennock, D.M., 1999. Aggregating probabilistic beliefs: Market mechanisms and graphical representations. PhD thesis. University of Michigan; Pennock, D.M., 1999. Aggregating probabilistic beliefs: Market mechanisms and graphical representations. PhD thesis. University of Michigan
[76] Pettit, P., Deliberative democracy and the discursive dilemma, Philosophical Issues, 11, 268-299 (2001)
[77] Predd, J. B., Kulkarni, S. R., Poor, H. V., Osherson, D., 2006. Scalable algorithms for aggregating disparate forecasts of probability. In: Ninth International Conference on Information Fusion; Predd, J. B., Kulkarni, S. R., Poor, H. V., Osherson, D., 2006. Scalable algorithms for aggregating disparate forecasts of probability. In: Ninth International Conference on Information Fusion
[78] Press, W. H.; Teukolsky, S. A.; Vetterling, W. T.; Flannery, B. P., Numerical Recipes in C: The Art of Scientific Computing (1992), Cambridge Univ. Press: Cambridge Univ. Press Cambridge, UK · Zbl 0845.65001
[79] Resnik, M. D., Choice: An Introduction to Decision Theory (1987), Univ. of Minnesota Press: Univ. of Minnesota Press Minneapolis, MN
[80] Rowe, G., Perspectives on expertise in aggregation of judgments, (Wright, G.; Bolger, P., Expertise and Decision Support (1992), Plenum Press: Plenum Press New York, NY), 155-180
[81] Schaller, M.; Asp, C. H.; Rosell, M. C.; Heim, S. J., Training in statistical reasoning inhibits formation of erroneous group stereotypes, Personality Soc. Psychol. Bull., 22, 829-844 (1996)
[82] Sides, A.; Osherson, D.; Bonini, N.; Viale, R., On the reality of the conjunction fallacy, Memory and Cognition, 30, 2, 191-198 (2002)
[83] Suppes, P., Probabilistic inference and the concept of total evidence, (Hintikka, J.; Suppes, P., Aspects of Inductive Logic (1966), North-Holland: North-Holland Amsterdam), 49-65 · Zbl 0202.29603
[84] Tentori, K.; Bonini, N.; Osherson, D., The conjunction fallacy: A misunderstanding about conjunction?, Cogn. Sci., 28, 3, 467-477 (2004)
[85] Vallone, R.; Ross, L.; Lepper, M., The hostile media phenomenon: Biased perception and perceptions of media bias in coverage of the Beirut massacre, J. Personality Soc. Psychol., 49, 577-585 (1985)
[86] van der Gaag, L.; Renooij, S.; Witteman, C.; Aleman, B.; Taal, B., How to elicit many probabilities, (Laskey, K. B.; Prade, H., Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence (1999), Morgan Kaufmann: Morgan Kaufmann San Francisco), 647-654
[87] van Laarhoven, P., 1988. Theoretical and computational aspects of simulated annealing. Center for Mathematics and Computer Science, Amsterdam; van Laarhoven, P., 1988. Theoretical and computational aspects of simulated annealing. Center for Mathematics and Computer Science, Amsterdam · Zbl 0659.90062
[88] von Winterfeldt, D.; Edwards, W., Decision Analysis and Behavioral Research (1986), Cambridge Univ. Press: Cambridge Univ. Press New York, NY
[89] Vose, M. D., The Simple Genetic Algorithm: Foundations and Theory (1999), MIT Press: MIT Press Cambridge, MA · Zbl 0952.65048
[90] Wagner, C., Aggregating subjective probabilities: Some limitative theorems, Notre Dame J. Formal Logic, 25, 3 (1984) · Zbl 0544.62010
[91] Walley, P., Statistical Reasoning with Imprecise Probabilities (1991), Chapman and Hall: Chapman and Hall London, England · Zbl 0732.62004
[92] Walley, P., Measures of uncertainty in expert systems, Artificial Intelligence, 83, 1-58 (1996) · Zbl 1506.68157
[93] Wallsten, T.; Budescu, D.; Erev, I.; Diederich, A., Evaluating and combining subjective probability estimates, J. Behav. Dec. Making, 10, 243-268 (1997)
[94] Winkler, R. L., The assessment of prior distributions in Bayesian analysis, J. Amer. Statist. Assoc., 62, 776-800 (1967)
[95] Winkler, R. L., Probabilistic prediction: Some experimental results, J. Amer. Statist. Assoc., 66, 675-685 (1971)
[96] Wright, G.; Ayton, P., The psychology of forecasting, (Wright, G.; Ayton, P., Judgmental Forecasting (1987), Wiley: Wiley New York, NY), 83-105
[97] Yates, J. F., Judgment and Decision Making (1990), Prentice Hall: Prentice Hall Englewood Cliffs, NJ
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.