Evaluating density forecasts: a comment

Tsyplakov, Alexander (2011): Evaluating density forecasts: a comment.

Abstract

This is a comment on Mitchell and Wallis (2011) which in turn is a critical reaction to Gneiting et al. (2007). The comment discusses the notion of forecast calibration, the advantage of using scoring rules, the “sharpness” principle and a general approach to testing calibration. The aim is to show how a more general and explicitly stated framework for evaluation of probabilistic forecasts can provide further insights.

• Item Type: MPRA Paper
• Language: English
• Keywords: density forecasts
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C53 - Forecasting and Prediction Methods ; Simulation Methods; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection

References

Bröcker, J. (2009). Reliability, sufficiency, and the decomposition of proper scores, Quarterly Journal of the Royal Meteorological Society 135(643): 1512–1519.

Clements, M. P. and Taylor, N. (2003). Evaluating interval forecasts of high-frequency financial data, Journal of Applied Econometrics 18(4): 445–456.

Corradi, V. and Swanson, N. R. (2006). Predictive density evaluation, in C. W. J. Granger, G. Elliott and A. Timmermann (eds), Handbook of Economic Forecasting, Vol. 1, North-Holland, Amsterdam, chapter 5, pp. 197–286.

DeGroot, M. H. (1962). Uncertainty, information, and sequential experiments, The Annals of Mathematical Statistics 33(2): 404–419.

Diebold, F. X., Gunther, T. A. and Tay, A. S. (1998). Evaluating density forecasts with applications to financial risk management, International Economic Review 39(4): 863–883.

Gneiting, T., Balabdaoui, F. and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness, Journal of the Royal Statistical Society: Series B 69: 243–268.

Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation, Journal of the American Statistical Association 102: 359–378.

Granger, C. W. J. and Pesaran, M. H. (2000). A decision-theoretic approach to forecast evaluation, in W.-S. Chan, W. K. Li and H. Tong (eds), Statistics and Finance: An Interface, Imperial College Press.

Lichtenstein, S., Fischhoff, B. and Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980, in D. Kahneman, P. Slovic and A. Tversky (eds), Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press, Cambridge, UK, pp. 306–334.

Mitchell, J. and Wallis, K. F. (2011). Evaluating density forecasts: Forecast combinations, model mixtures, calibration and sharpness, Journal of Applied Econometrics, forthcoming.

Rosenblatt, M. (1952). Remarks on a multivariate transformation, Annals of Mathematical Statistics 23: 470–472.

Sanders, F. (1963). On subjective probability forecasting, Journal of Applied Meteorology 2: 191–201.