Theoretical guidelines for a partially informed forecast examiner

Tsyplakov, Alexander (2014): Theoretical guidelines for a partially informed forecast examiner.

Abstract

The paper explores probability theory foundations behind evaluation of probabilistic forecasts. The emphasis is on a situation when the forecast examiner possesses only partially the information which was available and was used to produce a forecast. We argue that in such a situation forecasts should be judged by their conditional auto-calibration. Necessary and sufficient conditions of auto-calibration are discussed and expressed in the form of testable moment conditions. The paper also analyzes relationships between forecast calibration and forecast efficiency.

• Item Type: MPRA Paper
• Language: English
• Keywords: probabilistic forecast; forecast calibration; moment condition; probability integral transform; orthogonality condition; scoring rule; forecast encompassing
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C53 - Forecasting and Prediction Methods ; Simulation Methods

References

Amisano, G., and R. Giacomini (2007): “Comparing Density Forecasts via Weighted Likelihood Ratio Tests,” Journal of Business and Economic Statistics, 25(2), 177–190.

Bao, Y., T.-H. Lee, and B. Y. Saltoglu (2007): “Comparing Density Forecast Models,” Journal of Forecasting, 26, 203–225.

Berkowitz, J. (2001): “Testing Density Forecasts, With Applications to Risk Management,” Journal of Business & Economic Statistics, 19(4), 465–474.

Bierens, H. J.(2004): Introduction to the Mathematical and Statistical Foundations of Econometrics. Cambridge University Press.

Boero, G., J. Smith, and K. F. Wallis (2011): “Scoring Rules and Survey Density Forecasts,” International Journal of Forecasting, 27(2), 379–393.

Bollerslev, T. (1987): “A Conditionally Heteroskedastic Time Series Model for Speculative Prices and Rates of Return,” Review of Economics and Statistics, 69(3), 542–547.

Brocker, J. (2009): “Reliability, Sufficiency, and the Decomposition of Proper Scores,” Quarterly Journal of the Royal Meteorological Society, 135(643), 1512–1519.

Brocker, J., and L. A. Smith (2007): “Scoring Probabilistic Forecasts: The Importance of Being Proper,” Weather and Forecasting, 22, 382–388.

Brockwell, A. E.(2007): “Universal Residuals: A Multivariate Transformation,” Statistics & Probability Letters, 77, 1473–1478.

Chen, Y.-T. (2011): “Moment Tests for Density Forecast Evaluation in the Presence of Parameter Estimation Uncertainty,” Journal of Forecasting, 30(4), 409–450.

Chong, Y. Y., and D. F. Hendry (1986): “Econometric Evaluation of Linear Macro-Economic Models,” Review of Economic Studies, 53(4), 671–690.

Christoffersen, P. F. (1998): “Evaluating Interval Forecasts,” International Economic Review, 39(4), 841–862.

Clements, M. P. (2006): “Evaluating the Survey of Professional Forecasters Probability Distributions of Expected Inflation Based on Derived Event Probability Forecasts,” Empirical Economics, 31(1), 49–64.

Clements, M. P., and D. I. Harvey (2010): “Forecast Encompassing Tests and Probability Forecasts,” Journal of Applied Econometrics, 25(6), 1028–1062.

Clements, M. P., and N. Taylor (2003): “Evaluating Interval Forecasts of High-Frequency Financial Data,” Journal of Applied Econometrics, 18(4), 445–456.

Corradi, V., and N. R. Swanson (2005): “A Test for Comparing Multiple Misspecified Conditional Interval Models,” Econometric Theory, 21(05), 991–1016.

Corradi, V., and N. R. Swanson (2006a): “Bootstrap Conditional Distribution Tests in the Presence of Dynamic Misspecification,” Journal of Econometrics, 133(2), 779–806.

Corradi, V., and N. R. Swanson (2006b): “Predictive Density and Conditional Confidence Interval Accuracy Tests,” Journal of Econometrics, 135(1-2), 187–228.

Corradi, V., and N. R. Swanson (2006c): “Predictive Density Evaluation,” in Handbook of Economic Forecasting, ed. by C. W. J. Granger, G. Elliott, and A. Timmermann, vol. 1, chap. 5, pp. 197–286. North-Holland, Amsterdam.

Cox, D. R. (1961): “Tests of Separate Families of Hypotheses,” in Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability, vol.1, pp.105–123, Berkeley. University of California Press.

Cox, D. R. (1962): “Further Results on Tests of Separate Families of Hypotheses,” Journal of the Royal Statistical Society. Series B (Methodological), 24(2), 406–424.

Dawid, A. P. (1984): “Statistical Theory: The Prequential Approach,” Journal of the Royal Statistical Society. Series A (General), 147(2), 278–292.

DeGroot, M. H. (1962): “Uncertainty, Information, and Sequential Experiments,” The Annals of Mathematical Statistics, 33(2), 404–419.

DeGroot, M. H., and S. E. Fienberg (1983): “The Comparison and Evaluation of Forecasters,” Journal of the Royal Statistical Society. Series D (The Statistician), 32(1/2), 12–22.

Diebold, F. X., T. A. Gunther, and A. S. Tay (1998): “Evaluating Density Forecasts with Applications to Financial Risk Management,” International Economic Review, 39(4), 863–883.

Diebold, F. X., J. Hahn, and A. S. Tay (1999): “Multivariate Density Forecast Evaluation and Calibration in Financial Risk Management: High-Frequency Returns on Foreign Exchange,” Review of Economics and Statistics, 81(4), 661–673.

Diebold, F. X., and R. S. Mariano (1995): “Comparing Predictive Accuracy,” Journal of Business & Economic Statistics, 13(3), 253–263.

Diebold, F. X., and G. D. Rudebusch (1989): “Scoring the Leading Indicators,” The Journal of Business, 62(3), 369–391.

Diebold, F. X., A. S. Tay, and K. F. Wallis (1999): “Evaluating Density Forecasts of Inflation: The Survey of Professional Forecasters,” in Cointegration, Causality and Forecasting: A Festschrift in Honour of Clive Granger, ed. By R F. Engle, and H. White, pp. 76–90 Oxford University Press, Oxford.

Diks, C., V. Panchenko, and D. van Dijk (2011): “Likelihood-Based Scoring Rules for Comparing Density Forecasts in Tails,” Journal of Econometrics, 163(2), 215–230.

Engelberg, J., C. F. Manski, and J. Williams (2009): “Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters,” Journal of Business and Economic Statistics, 27(1), 30–41.

Engle, R. F., and S. Manganelli (2004): “CAViaR,” Journal of Business & Economic Statistics, 22(4), 367–381.

Ferguson, T. S. (1967): Mathematical Statistics: A Decision Theoretic Approach. Academic Press, New York.

Galbraith, J. W., and S. van Norden (2011): “Kernel-Based Calibration Diagnostics for Recession and Inflation Probability Forecasts,” International Journal of Forecasting, 27(4), 1041–1057.

Giacomini, R., and H. White (2006): “Tests of Conditional Predictive Ability,” Econometrica, 74(6), 1545–1578.

Gneiting, T., F. Balabdaoui, and A. E. Raftery (2007): “Probabilistic Forecasts, Calibration and Sharpness,” Journal of the Royal Statistical Society: Series B, 69, 243–268.

Gneiting, T., and A. E. Raftery (2007): “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, 102, 359–378.

Gneiting, T., and R. Ranjan (2013): “Combining Predictive Distributions,” Electronic Journal of Statistics, 7, 1747–1782.

Granger,C.W.J.(1999): “Outline of Forecast Theory Using Generalized Cost Functions,” Spanish Economic Review, 1, 161–173.

Granger, C. W. J., and M. H. Pesaran (2000): “A Decision-Theoretic Approach to Forecast Evaluation,” in Statistics and Finance: An Interface, ed. by W.-S. Chan, W. K. Li, and H. Tong. Imperial College Press.

Hall, S. G., and J. Mitchell (2007): “Combining Density Forecasts,” International Journal of Forecasting, 23, 1–13.

Hansen, L. P. (1982): “Large Sample Properties of Generalized Method of Moments Estimators,” Econometrica, 50(4), 1029–1054.

Holzmann, H., and M. Eulert (2011): “The Role of the Information Set for Forecasting — With Applications to Risk Management,” (unpublished).

Kallenberg, O. (2002): Foundations of Modern Probability. Springer, 2 edn.

Kupiec, P. H. (1995): “Techniques for Verifying the Accuracy of Risk Measurement Models,” Journal of Derivatives, 3(2), 73–84.

Lopez, J. A. (1998): “Methods for Evaluating Value-at-Risk Estimates,” Economic Policy Review, (October), 119–124.

Mincer, J. A., and V. Zarnowitz (1969): “The Evaluation of Economic Forecasts,” in Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, ed. by J. A. Mincer, pp. 3–46. National Bureau of Economic Research.

Mitchell, J., and K. F. Wallis (2011): “Evaluating Density Forecasts: Forecast Combinations, Model Mixtures, Calibration and Sharpness,” Journal of Applied Econometrics, 26(6), 1023– 1040.

Murphy, A. H., and R. L. Winkler (1987): “A General Framework for Forecast Verification,” Monthly Weather Review, 115, 1330–1338.

Pesaran, M. H., and S. Skouras (2002): “Decision-Based Methods for Forecast Evaluation,” in A Companion to Economic Forecasting, ed. by M. P. Clements, and D. F. Hendry, chap. 11, pp. 241–267. Blackwell.

RiskMetrics (1996): “RiskMetrics(TM) — Technical Document (4th ed.),” Discussion paper, J. P. Morgan/Reuters‘.

Sanders, F. (1963): “On Subjective Probability Forecasting,” Journal of Applied Meteorology, 2, 191–201.

Sarno, L., and G. Valente (2004): “Comparing the Accuracy of Density Forecasts from Competing Models,” Journal of Forecasting, 23, 541–557.

Shiller, R. J. (1978): “Rational Expectations and the Dynamic Structure of Macroeconomic Models: A Critical Review,” Journal of Monetary Economics, 4(1), 1–44.

Tsyplakov, A. (2011): “Evaluating Density Forecasts: A Comment,” MPRA Paper 32728, University Library of Munich, Germany.

Wallis, K. F. (2003): “Chi-Squared Tests of Interval and Density Forecasts, and the Bank of England’s Fan Charts,” International Journal of Forecasting, 19(2), 165–175.

West, K. D. (1996): “Asymptotic Inference about Predictive Ability,” Econometrica, 64, 1067– 1087.

White, H. (2000): “A Reality Check for Data Snooping,” Econometrica, 68(5), 1097–1126.