Theoretical guidelines for a partially informed forecast examiner

Tsyplakov, Alexander (2014): Theoretical guidelines for a partially informed forecast examiner.

Abstract

The paper explores theoretical foundations behind evaluation of probabilistic forecasts. The emphasis is on a situation when the forecast examiner possesses only partially the information which was available and was used to produce a forecast. We argue that in such a situation forecasts should be judged by their conditional auto-calibration. Necessary and sufficient conditions of auto-calibration are discussed and expressed in the form of testable moment conditions. The paper also analyzes relationships between forecast calibration and forecast efficiency.

• Item Type: MPRA Paper
• Language: English
• Keywords: probabilistic forecast; forecast calibration; moment condition; probability integral transform; orthogonality condition; scoring rule; forecast encompassing
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C53 - Forecasting and Prediction Methods ; Simulation Methods

References

Barndorff-Nielsen, O. E., and D. R. Cox (1996): “Prediction and Asymptotics,” Bernoulli, 2(4), 319–340.

Berkowitz, J. (2001): “Testing Density Forecasts, With Applications to Risk Management,” Journal of Business & Economic Statistics, 19(4), 465–474.

Bierens, H. J. (2004): Introduction to the Mathematical and Statistical Foundations of Econometrics. Cambridge University Press.

Boero, G., J. Smith, and K. F. Wallis (2011): “Scoring Rules and Survey Density Forecasts,” International Journal of Forecasting, 27(2), 379–393.

Bröcker, J. (2009): “Reliability, Sufficiency, and the Decomposition of Proper Scores,” Quarterly Journal of the Royal Meteorological Society, 135(643), 1512–1519.

Bröcker, J., and L. A. Smith (2007): “Scoring Probabilistic Forecasts: The Importance of Being Proper,” Weather and Forecasting, 22, 382–388.

Brockwell, A. E. (2007): “Universal Residuals: A Multivariate Transformation,” Statistics & Probability Letters, 77, 1473–1478.

Chen, Y.-T. (2011): “Moment Tests for Density Forecast Evaluation in the Presence of Parameter Estimation Uncertainty,” Journal of Forecasting, 30(4), 409–450.

Chong, Y. Y., and D. F. Hendry (1986): “Econometric Evaluation of Linear Macro-Economic Models,” Review of Economic Studies, 53(4), 671–690.

Christoffersen, P. F. (1998): “Evaluating Interval Forecasts,” International Economic Review, 39(4), 841–862.

Clements, M. P. (2006): “Evaluating the Survey of Professional Forecasters Probability Distributions of Expected Inflation Based on Derived Event Probability Forecasts,” Empirical Economics, 31(1), 49–64.

Clements, M. P., and D. I. Harvey (2010): “Forecast Encompassing Tests and Probability Forecasts,” Journal of Applied Econometrics, 25(6), 1028–1062.

Clements, M. P., and N. Taylor (2003): “Evaluating Interval Forecasts of High-Frequency Financial Data,” Journal of Applied Econometrics, 18(4), 445–456.

Cooley, T. F., and W. R. Parke (1990): “Asymptotic Likelihood-Based Prediction Functions,” Econometrica, 58(5), 1215–1234.

Corradi, V., and N. R. Swanson (2006a): “Bootstrap Conditional Distribution Tests in the Presence of Dynamic Misspecification,” Journal of Econometrics, 133(2), 779–806.

Corradi, V., and N. R. Swanson (2006b): “Predictive Density and Conditional Confidence Interval Accuracy Tests,” Journal of Econometrics, 135(1-2), 187–228.

Corradi, V., and N. R. Swanson (2006c): “Predictive Density Evaluation,” in Handbook of Economic Forecasting, ed. by C. W. J.

Granger, G. Elliott, and A. Timmermann, vol. 1, chap. 5, pp. 197–286. North-Holland, Amsterdam.

Cox,D.R.(1961): “Tests of Separate Families of Hypotheses,” in Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 105–123, Berkeley. University of California Press.

Cox, D. R. (1962): “Further Results on Tests of Separate Families of Hypotheses,” Journal of the Royal Statistical Society. Series B (Methodological), 24(2), 406–424.

Dawid, A. P. (1984): “Statistical Theory: The Prequential Approach,” Journal of the Royal Statistical Society. Series A (General), 147(2), 278–292.

DeGroot, M. H. (1962): “Uncertainty, Information, and Sequential Experiments,” The Annals of Mathematical Statistics, 33(2), 404–419.

DeGroot, M. H., and S. E. Fienberg (1983): “The Comparison and Evaluation of Forecasters,” Journal of the Royal Statistical Society. Series D (The Statistician), 32(1/2), 12–22.

Diebold, F. X., T. A. Gunther, and A. S. Tay (1998): “Evaluating Density Forecasts with Applications to Financial Risk Management,” International Economic Review, 39(4), 863–883.

Diebold, F. X., J. Hahn, and A. S. Tay (1999): “Multivariate Density Forecast Evaluation and Calibration in Financial Risk Management: High-Frequency Returns on Foreign Exchange,” Review of Economics and Statistics, 81(4), 661–673.

Diebold, F. X., and G. D. Rudebusch (1989): “Scoring the Leading Indicators,” The Journal of Business, 62(3), 369–391.

Diebold, F. X., A. S. Tay, and K. F. Wallis (1999): “Evaluating Density Forecasts of Inflation: The Survey of Professional Forecasters,” in Cointegration, Causality and Forecasting: A Festschrift in Honour of Clive

Granger, ed. by R. F. Engle, and H. White, pp. 76–90. Oxford University Press, Oxford.

Diks, C., V. Panchenko, and D. van Dijk (2011): “Likelihood-Based Scoring Rules for Comparing Density Forecasts in Tails,” Journal of Econometrics, 163(2), 215–230.

Engelberg, J., C. F. Manski, and J. Williams (2009): “Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters,” Journal of Business and Economic Statistics, 27(1), 30–41.

Engle, R. F., and S. Manganelli (2004): “CAViaR,” Journal of Business & Economic Statistics, 22(4), 367–381.

Ferguson, T. S.(1967): Mathematical Statistics: A Decision Theoretic Approach. Academic Press, New York.

Galbraith, J. W., and S. van Norden (2011): “Kernel-Based Calibration Diagnostics for Recession and Inflation Probability Forecasts,” International Journal of Forecasting, 27(4), 1041–1057.

Geweke, J., and G. Amisano (2011): “Optimal Prediction Pools,” Journal of Econometrics, 164(1), 130–141.

Giacomini, R., and H. White (2006): “Tests of Conditional Predictive Ability,” Econometrica, 74(6), 1545–1578.

Gneiting,T.,F.Balabdaoui, and A.E.Raftery(2007): “Probabilistic Forecasts, Calibration and Sharpness,” Journal of the Royal Statistical Society: Series B, 69, 243–268.

Gneiting, T., and A.E.Raftery (2007): “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, 102, 359–378.

Gneiting, T., and R. Ranjan (2013): “Combining Predictive Distributions,” Electronic Journal of Statistics, 7, 1747–1782.

Granger, C. W. J. (1999): “Outline of Forecast Theory Using Generalized Cost Functions,” Spanish Economic Review, 1, 161–173.

Granger, C. W. J., and M. H. Pesaran (2000): “A Decision-Theoretic Approach to Forecast Evaluation,” in Statistics and Finance: An Interface, ed. by W.-S. Chan, W. K. Li, and H. Tong. Imperial College Press.

Hansen, L. P. (1982): “Large Sample Properties of Generalized Method of Moments Estimators,” Econometrica, 50(4), 1029–1054.

Holzmann,H., and M.Eulert (2014): “The Role of the Information Set for Forecasting—With Applications to Risk Management,” The Annals of Applied Statistics, 8(1), 595–621.

Kallenberg, O. (2002): Foundations of Modern Probability. Springer, 2 edn.

Knüppel, M. (2015): “Evaluating the Calibration of Multi-Step-Ahead Density Forecasts Using Raw Moments,” Journal of Business and Economic Statistics, 33(2), 270–281.

Kupiec, P. H. (1995): “Techniques for Verifying the Accuracy of Risk Measurement Models,” Journal of Derivatives, 3(2), 73–84.

Lahiri, K., and L. Yang (2015): “A Further Analysis of the Conference Board’s New Leading Economic Index,” International Journal of Forecasting, 31(2), 446–453.

Lichtenstein, S., B. Fischhoff, and L. D. Phillips (1982): “Calibration of Probabilities: The State of the Art to 1980,” in Judgment under Uncertainty: Heuristics and Biases, ed. by D. Kahneman, P. Slovic, and A. Tversky, pp. 306–334. Cambridge University Press, Cambridge, UK.

Lopez, J.A.(1998): “Methods for Evaluating Value-at-Risk Estimates,” Economic Policy Review, (October), 119–124.

Mincer, J. A., and V. Zarnowitz (1969): “The Evaluation of Economic Forecasts,” in Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, ed. by J. A. Mincer, pp. 3–46. National Bureau of Economic Research.

Mitchell, J., and K. F. Wallis (2011): “Evaluating Density Forecasts: Forecast Combinations, Model Mixtures, Calibration and Sharpness,” Journal of Applied Econometrics,, 26(6), 1023–1040.

Murphy, A. H., and R. L. Winkler (1987): “A General Framework for Forecast Verification,” Monthly Weather Review, 115, 1330–1338.

Pesaran, M. H., and S. Skouras (2002): “Decision-Based Methods for Forecast Evaluation,” in A Companion to Economic Forecasting, ed. by M. P. Clements, and D. F. Hendry, chap. 11, pp. 241–267. Blackwell.

Sanders, F. (1963): “On Subjective Probability Forecasting,” Journal of Applied Meteorology, 2, 191–201.

Shiller, R. J. (1978): “Rational Expectations and the Dynamic Structure of Macroeconomic Models: A Critical Review,” Journal of Monetary Economics, 4(1), 1–44.

The National Institute of Economic Research (2013): “The Riksbank Has Systematically Overestimated Inflation,” The Swedish Economy, pp. 29–33.

Tsyplakov, A.(2011): “Evaluating Density Forecasts: A Comment,” MPRA Paper 32728, University Library of Munich, Germany.

Wallis, K. F. (2003): “Chi-Squared Tests of Interval and Density Forecasts, and the Bank of England’s Fan Charts,” International Journal of Forecasting, 19(2), 165–175.