Evaluation of Probabilistic Forecasts: Proper Scoring Rules and Moments

Tsyplakov, Alexander (2013): Evaluation of Probabilistic Forecasts: Proper Scoring Rules and Moments.

Abstract

The paper provides an overview of probabilistic forecasting and discusses a theoretical framework for evaluation of probabilistic forecasts which is based on proper scoring rules and moments. An artificial example of predicting second-order autoregression and an example of predicting the RTSI stock index are used as illustrations.

• Item Type: MPRA Paper
• Language: English
• Keywords: probabilistic forecast; forecast calibration; probability integral transform; scoring rule; moment condition
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C53 - Forecasting and Prediction Methods ; Simulation Methods

References

Amisano, G. and Giacomini, R. (2007). Comparing density forecasts via weighted likelihood ratio tests, Journal of Business and Economic Statistics 25(2): 177–190.

Berkowitz, J. (2001). Testing density forecasts, with applications to risk management, Journal of Business & Economic Statistics 19(4): 465–474.

Bhattacharya, R. and Waymire, E. C. (2007). A Basic Course in Probability Theory, Springer.

Bierens, H. J. (2004). Introduction to the Mathematical and Statistical Foundations of Econometrics, Cambridge University Press.

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability, Monthly Weather Review 78(1): 1–3.

Britton, E., Fisher, P. and Whitley, J. (1998). The Inflation Report projections: Understanding the fan chart, Bank of England Quarterly Bulletin 38: 30–37.

Bröcker, J. (2009). Reliability, sufficiency, and the decomposition of proper scores, Quarterly Journal of the Royal Meteorological Society 135(643): 1512–1519.

Bröcker, J. and Smith, L. A. (2007). Scoring probabilistic forecasts: The importance of being proper, Weather and Forecasting 22: 382–388.

Brockwell, A. E. (2007). Universal residuals: A multivariate transformation, Statistics & Probability Letters 77: 1473–1478.

Christoffersen, P. F. (1998). Evaluating interval forecasts, International Economic Review 39(4): 841–862.

Clements, M. P. (2004). Evaluating the Bank of England density forecasts of inflation, The Economic Journal 114: 844–866.

Clements, M. P. and Taylor, N. (2003). Evaluating interval forecasts of high-frequency financial data, Journal of Applied Econometrics 18(4): 445–456.

DeGroot, M. H. (1962). Uncertainty, information, and sequential experiments, The Annals of Mathematical Statistics 33(2): 404–419.

DeGroot, M. H. and Fienberg, S. E. (1983). The comparison and evaluation of forecasters, Journal of the Royal Statistical Society. Series D (The Statistician) 32(1/2): 12–22.

Diebold, F. X., Gunther, T. A. and Tay, A. S. (1998). Evaluating density forecasts with applications to financial risk management, International Economic Review 39(4): 863–883.

Diebold, F. X., Hahn, J. and Tay, A. S. (1999). Multivariate density forecast evaluation and calibration in financial risk management: High-frequency returns on foreign exchange, Review of Economics and Statistics 81(4): 661–673.

Diebold, F. X., Tay, A. S. and Wallis, K. F. (1999). Evaluating density forecasts of inflation: The survey of professional forecasters, in R. F. Engle and H. White (eds), Cointegration, Causality and Forecasting: A Festschrift in Honour of Clive Granger, Oxford University Press, Oxford, pp. 76–90.

Engelberg, J., Manski, C. F. and Williams, J. (2009). Comparing the point predictions and subjective probability distributions of professional forecasters, Journal of Business and Economic Statistics 27(1): 30–41.

Galbraith, J. W. and van Norden, S. (2011). Kernel-based calibration diagnostics for recession and inflation probability forecasts, International Journal of Forecasting 27(4): 1041–1057.

Gneiting, T. (2011). Making and evaluating point forecasts, Journal of the American Statistical Association 106(494): 746–762.

Gneiting, T., Balabdaoui, F. and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness, Journal of the Royal Statistical Society: Series B 69: 243–268.

Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation, Journal of the American Statistical Association 102: 359–378.

Gneiting, T. and Ranjan, R. (2011). Combining predictive distributions, ArXiv preprint. arXiv:1106.1638v1 [math.ST].

Granger, C. W. J. (1999). Outline of forecast theory using generalized cost functions, Spanish Economic Review 1: 161–173.

Granger, C. W. J. and Pesaran, M. H. (2000). A decision-theoretic approach to forecast evaluation, in W.-S. Chan, W. K. Li and H. Tong (eds), Statistics and Finance: An Interface, Imperial College Press.

Holzmann, H. and Eulert, M. (2011). The role of the information set for forecasting — with applications to risk management. (unpublished).

Lichtenstein, S., Fischhoff, B. and Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980, in D. Kahneman, P. Slovic and A. Tversky (eds), Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press, Cambridge, UK, pp. 306–334.

Mitchell, J. and Wallis, K. F. (2011). Evaluating density forecasts: Forecast combinations, model mixtures, calibration and sharpness, Journal of Applied Econometrics, 26(6): 1023–1040.

Murphy, A. H. and Winkler, R. L. (1984). Probability forecasting in meteorology, Journal of the American Statistical Association 79(387): 489–500.

National Research Council (U.S.) (2006). Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts, National Academies Press.

Pesaran, M. H. and Skouras, S. (2002). Decision-based methods for forecast evaluation, in M. P. Clements and D. F. Hendry (eds), A Companion to Economic Forecasting, Blackwell Publishing, chapter 11, pp. 241–267.

RiskMetrics (1996). Riskmetrics(TM) — Technical document (4th ed.), Technical report, J. P. Morgan/Reuters‘.

Sanders, F. (1963). On subjective probability forecasting, Journal of Applied Meteorology 2: 191–201.

Shiller, R. J. (1978). Rational expectations and the dynamic structure of macroeconomic models: A critical review, Journal of Monetary Economics 4(1): 1–44.