×

zbMATH — the first resource for mathematics

Probabilistic forecasts, calibration and sharpness. (English) Zbl 1120.62074
Summary: Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only.
A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with cross-validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.

MSC:
62M20 Inference from stochastic processes and prediction
62P12 Applications of statistics to environmental and related topics
62J20 Diagnostics, and linear inference and regression
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] DOI: 10.1175/1520-0442(1996)009<1518:AMFPAE>2.0.CO;2 · doi:10.1175/1520-0442(1996)009<1518:AMFPAE>2.0.CO;2
[2] DOI: 10.1016/j.ijforecast.2003.09.014 · doi:10.1016/j.ijforecast.2003.09.014
[3] DOI: 10.1198/07350010152596718 · Zbl 04569369 · doi:10.1198/07350010152596718
[4] Bernardo J. M., Ann. Statist. 7 pp 686– (1979)
[5] Besag J., Statist. Sci. 10 pp 3– (1995)
[6] Blum J. R., Z. Wahrsch. Ver. Geb. 2 pp 1– (1963)
[7] Boero G., Int. J. Forecast. 20 pp 305– (2004)
[8] DOI: 10.1175/1520-0493(2004)132<0338:PFOPIT>2.0.CO;2 · doi:10.1175/1520-0493(2004)132<0338:PFOPIT>2.0.CO;2
[9] Brier G. W., Mnthly Weath. Rev. 78 pp 1– (1950)
[10] Brocklehurst S., Handbook of Software Reliability Engineering (1995)
[11] Brown B. G., J. Clim. Appl. Meteorol. 23 pp 1184– (1984)
[12] DOI: 10.1256/qj.04.71 · doi:10.1256/qj.04.71
[13] Christoffersen P. F., Int. Econ. Rev. 39 pp 841– (1998)
[14] DOI: 10.1002/1099-131X(200007)19:4<255::AID-FOR773>3.0.CO;2-G · doi:10.1002/1099-131X(200007)19:4<255::AID-FOR773>3.0.CO;2-G
[15] Corradi V., Handbook of Economic Forecasting, vol. 1 pp 197– (2006)
[16] DOI: 10.2307/2287720 · Zbl 0495.62005 · doi:10.2307/2287720
[17] Dawid A. P., J. R. Statist. Soc. 147 pp 278– (1984)
[18] Dawid A. P., J. Am. Statist. Ass. 80 pp 340– (1985)
[19] Dawid A. P., Ann. Statist. 13 pp 1251– (1985)
[20] Dawid A. P., Encyclopedia of Statistical Sciences, vol. 7 pp 210– (1986)
[21] Dawid A. P., Bernoulli 5 pp 125– (1999)
[22] DeGroot M. H., Statistical Decision Theory and Related Topics III pp 291– (1982)
[23] DeGroot M. H., Statistician 12 pp 12– (1983)
[24] DOI: 10.2307/2527342 · doi:10.2307/2527342
[25] DOI: 10.2307/1392185 · doi:10.2307/1392185
[26] Duffie D., J. Deriv. 4 pp 7– (1997)
[27] DOI: 10.1093/biomet/85.2.379 · Zbl 0947.62059 · doi:10.1093/biomet/85.2.379
[28] Fruhwirth-Schnatter S., Environ. Ecol. Statist. 3 pp 291– (1996)
[29] DOI: 10.1198/016214503000000765 · doi:10.1198/016214503000000765
[30] Gelman A., Statist. Sin. 6 pp 733– (1996)
[31] T. Gerds (2002 ) Nonparametric efficient estimation of prediction error for incomplete data models .PhD Thesis. Mathematische Fakultat, Albert-Ludwigs-Universitat Freiburg, Freiburg. · Zbl 1042.62530
[32] T. Gneiting, K. Larson, K. Westrick, M. G. Genton, and E. Aldrich (2004 ) Calibrated probabilistic forecasting at the Stateline wind energy centre: the regime-switching space-time (RST) method.Technical Report 464. Department of Statistics, University of Washington, Seattle. · Zbl 1120.62341
[33] DOI: 10.1198/016214506000000456 · Zbl 1120.62341 · doi:10.1198/016214506000000456
[34] Gneiting T., Science 310 pp 248– (2005)
[35] Gneiting T., J. Am. Statist. Ass. (2006)
[36] DOI: 10.1175/MWR2904.1 · doi:10.1175/MWR2904.1
[37] Good I. J., J. R. Statist. Soc. B 14 pp 107– (1952)
[38] Granger C. W. J., Oxf. Bull. Econ. Statist. 67 pp 707– (2006)
[39] S. Gschloss l, and C. Czado (2005 ) Spatial modelling of claim frequency and claim size in insurance .Discussion Paper 461. Sonderforschungsbereich 386, Ludwig-Maximilians-Universitat Munchen, Munich.
[40] DOI: 10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2 · doi:10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2
[41] Hamill T. M., Mnthly Weath. Rev. 125 pp 1312– (1997)
[42] J. Hoeting (1994 ) Accounting for model uncertainty in linear regression .PhD Thesis. Department of Statistics, University of Washington, Seattle.
[43] Jolliffe I. T., Forecast Verification: a Practitioner’s Guide in Atmospheric Science (2003)
[44] Krzysztofowicz R., Wat. Resour. Res. 35 pp 2739– (1999)
[45] Krzysztofowicz R., Weath. Forecast. 14 pp 427– (1999)
[46] Moyeed R. A., Math. Geol. 34 pp 365– (2002)
[47] Murphy A. H., J. Appl. Meteorol. 11 pp 273– (1972)
[48] Murphy A. H., Weath. Forecast. 4 pp 485– (1989)
[49] DOI: 10.1175/1520-0493(1987)115<1330:AGFFFV>2.0.CO;2 · doi:10.1175/1520-0493(1987)115<1330:AGFFFV>2.0.CO;2
[50] DOI: 10.1016/0169-2070(92)90028-8 · doi:10.1016/0169-2070(92)90028-8
[51] Noceti P., J. Forecast. 22 pp 447– (2003)
[52] Oakes D., J. Am. Statist. Ass. 80 pp 339– (1985)
[53] DOI: 10.1256/0035900021643593 · doi:10.1256/0035900021643593
[54] DOI: 10.2307/2332290 · doi:10.2307/2332290
[55] DOI: 10.1175/MWR2906.1 · doi:10.1175/MWR2906.1
[56] DOI: 10.2307/2291462 · Zbl 0888.62026 · doi:10.2307/2291462
[57] Rosenblatt M., Ann. Math. Statist. 23 pp 470– (1952)
[58] Roulston M. S., Mnthly Weath. Rev. 130 pp 1653– (2002)
[59] Roulston M. S., Tellus A 55 pp 16– (2003)
[60] Rubin D. B., Ann. Statist. 12 pp 1151– (1984)
[61] DOI: 10.1287/moor.28.1.141.14264 · Zbl 1082.90544 · doi:10.1287/moor.28.1.141.14264
[62] Schervish M. J., J. Am. Statist. Ass. 80 pp 341– (1985)
[63] Schervish M. J., Ann. Statist. 17 pp 1856– (1989)
[64] Schumacher M., Meth. Inform. Med. 42 pp 564– (2003)
[65] Seillier-Moiseiwitsch F., Int. Statist. Rev. 61 pp 395– (1993)
[66] DOI: 10.1023/A:1009957816843 · Zbl 0919.90003 · doi:10.1023/A:1009957816843
[67] Shafer G., Probability and Finance: It’s Only a Game! (2001) · Zbl 0985.91024 · doi:10.1002/0471249696
[68] DOI: 10.2307/2337055 · doi:10.2307/2337055
[69] Smith J. Q., J. Forecast. 4 pp 283– (1985)
[70] Stael von Holstein C.-A. S., Assessment and Evaluation of Subjective Probability Distributions (1970)
[71] Talagrand O., Proc. Wrkshp Predictability pp 1– (1997)
[72] DOI: 10.1111/j.1467-9868.2005.00525.x · Zbl 1112.62311 · doi:10.1111/j.1467-9868.2005.00525.x
[73] DOI: 10.1016/S0169-2070(02)00009-2 · doi:10.1016/S0169-2070(02)00009-2
[74] Weigend A. S., J. Forecast. 19 pp 375– (2000)
[75] Winkler R. L., Decision Making and Change in Human Affairs pp 127– (1977) · doi:10.1007/978-94-010-1276-8_10
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.