Tsyplakov, Alexander (2014): Theoretical guidelines for a partially informed forecast examiner.
There is a more recent version of this item available. 

PDF
MPRA_paper_55017.pdf Download (334kB)  Preview 
Abstract
The paper explores probability theory foundations behind evaluation of probabilistic forecasts. The emphasis is on a situation when the forecast examiner possesses only partially the information which was available and was used to produce a forecast. We argue that in such a situation forecasts should be judged by their conditional autocalibration. Necessary and sufficient conditions of autocalibration are discussed and expressed in the form of testable moment conditions. The paper also analyzes relationships between forecast calibration and forecast efficiency.
Item Type:  MPRA Paper 

Original Title:  Theoretical guidelines for a partially informed forecast examiner 
Language:  English 
Keywords:  probabilistic forecast; forecast calibration; moment condition; probability integral transform; orthogonality condition; scoring rule; forecast encompassing 
Subjects:  C  Mathematical and Quantitative Methods > C5  Econometric Modeling > C52  Model Evaluation, Validation, and Selection C  Mathematical and Quantitative Methods > C5  Econometric Modeling > C53  Forecasting and Prediction Methods ; Simulation Methods 
Item ID:  55017 
Depositing User:  Alexander Tsyplakov 
Date Deposited:  03. Apr 2014 10:59 
Last Modified:  03. Apr 2014 11:42 
References:  Amisano, G., and R. Giacomini (2007): “Comparing Density Forecasts via Weighted Likelihood Ratio Tests,” Journal of Business and Economic Statistics, 25(2), 177–190. Bao, Y., T.H. Lee, and B. Y. Saltoglu (2007): “Comparing Density Forecast Models,” Journal of Forecasting, 26, 203–225. Berkowitz, J. (2001): “Testing Density Forecasts, With Applications to Risk Management,” Journal of Business & Economic Statistics, 19(4), 465–474. Bierens, H. J.(2004): Introduction to the Mathematical and Statistical Foundations of Econometrics. Cambridge University Press. Boero, G., J. Smith, and K. F. Wallis (2011): “Scoring Rules and Survey Density Forecasts,” International Journal of Forecasting, 27(2), 379–393. Bollerslev, T. (1987): “A Conditionally Heteroskedastic Time Series Model for Speculative Prices and Rates of Return,” Review of Economics and Statistics, 69(3), 542–547. Brocker, J. (2009): “Reliability, Sufficiency, and the Decomposition of Proper Scores,” Quarterly Journal of the Royal Meteorological Society, 135(643), 1512–1519. Brocker, J., and L. A. Smith (2007): “Scoring Probabilistic Forecasts: The Importance of Being Proper,” Weather and Forecasting, 22, 382–388. Brockwell, A. E.(2007): “Universal Residuals: A Multivariate Transformation,” Statistics & Probability Letters, 77, 1473–1478. Chen, Y.T. (2011): “Moment Tests for Density Forecast Evaluation in the Presence of Parameter Estimation Uncertainty,” Journal of Forecasting, 30(4), 409–450. Chong, Y. Y., and D. F. Hendry (1986): “Econometric Evaluation of Linear MacroEconomic Models,” Review of Economic Studies, 53(4), 671–690. Christoffersen, P. F. (1998): “Evaluating Interval Forecasts,” International Economic Review, 39(4), 841–862. Clements, M. P. (2006): “Evaluating the Survey of Professional Forecasters Probability Distributions of Expected Inflation Based on Derived Event Probability Forecasts,” Empirical Economics, 31(1), 49–64. Clements, M. P., and D. I. Harvey (2010): “Forecast Encompassing Tests and Probability Forecasts,” Journal of Applied Econometrics, 25(6), 1028–1062. Clements, M. P., and N. Taylor (2003): “Evaluating Interval Forecasts of HighFrequency Financial Data,” Journal of Applied Econometrics, 18(4), 445–456. Corradi, V., and N. R. Swanson (2005): “A Test for Comparing Multiple Misspecified Conditional Interval Models,” Econometric Theory, 21(05), 991–1016. Corradi, V., and N. R. Swanson (2006a): “Bootstrap Conditional Distribution Tests in the Presence of Dynamic Misspecification,” Journal of Econometrics, 133(2), 779–806. Corradi, V., and N. R. Swanson (2006b): “Predictive Density and Conditional Confidence Interval Accuracy Tests,” Journal of Econometrics, 135(12), 187–228. Corradi, V., and N. R. Swanson (2006c): “Predictive Density Evaluation,” in Handbook of Economic Forecasting, ed. by C. W. J. Granger, G. Elliott, and A. Timmermann, vol. 1, chap. 5, pp. 197–286. NorthHolland, Amsterdam. Cox, D. R. (1961): “Tests of Separate Families of Hypotheses,” in Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability, vol.1, pp.105–123, Berkeley. University of California Press. Cox, D. R. (1962): “Further Results on Tests of Separate Families of Hypotheses,” Journal of the Royal Statistical Society. Series B (Methodological), 24(2), 406–424. Dawid, A. P. (1984): “Statistical Theory: The Prequential Approach,” Journal of the Royal Statistical Society. Series A (General), 147(2), 278–292. DeGroot, M. H. (1962): “Uncertainty, Information, and Sequential Experiments,” The Annals of Mathematical Statistics, 33(2), 404–419. DeGroot, M. H., and S. E. Fienberg (1983): “The Comparison and Evaluation of Forecasters,” Journal of the Royal Statistical Society. Series D (The Statistician), 32(1/2), 12–22. Diebold, F. X., T. A. Gunther, and A. S. Tay (1998): “Evaluating Density Forecasts with Applications to Financial Risk Management,” International Economic Review, 39(4), 863–883. Diebold, F. X., J. Hahn, and A. S. Tay (1999): “Multivariate Density Forecast Evaluation and Calibration in Financial Risk Management: HighFrequency Returns on Foreign Exchange,” Review of Economics and Statistics, 81(4), 661–673. Diebold, F. X., and R. S. Mariano (1995): “Comparing Predictive Accuracy,” Journal of Business & Economic Statistics, 13(3), 253–263. Diebold, F. X., and G. D. Rudebusch (1989): “Scoring the Leading Indicators,” The Journal of Business, 62(3), 369–391. Diebold, F. X., A. S. Tay, and K. F. Wallis (1999): “Evaluating Density Forecasts of Inflation: The Survey of Professional Forecasters,” in Cointegration, Causality and Forecasting: A Festschrift in Honour of Clive Granger, ed. By R F. Engle, and H. White, pp. 76–90 Oxford University Press, Oxford. Diks, C., V. Panchenko, and D. van Dijk (2011): “LikelihoodBased Scoring Rules for Comparing Density Forecasts in Tails,” Journal of Econometrics, 163(2), 215–230. Engelberg, J., C. F. Manski, and J. Williams (2009): “Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters,” Journal of Business and Economic Statistics, 27(1), 30–41. Engle, R. F., and S. Manganelli (2004): “CAViaR,” Journal of Business & Economic Statistics, 22(4), 367–381. Ferguson, T. S. (1967): Mathematical Statistics: A Decision Theoretic Approach. Academic Press, New York. Galbraith, J. W., and S. van Norden (2011): “KernelBased Calibration Diagnostics for Recession and Inflation Probability Forecasts,” International Journal of Forecasting, 27(4), 1041–1057. Giacomini, R., and H. White (2006): “Tests of Conditional Predictive Ability,” Econometrica, 74(6), 1545–1578. Gneiting, T., F. Balabdaoui, and A. E. Raftery (2007): “Probabilistic Forecasts, Calibration and Sharpness,” Journal of the Royal Statistical Society: Series B, 69, 243–268. Gneiting, T., and A. E. Raftery (2007): “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, 102, 359–378. Gneiting, T., and R. Ranjan (2013): “Combining Predictive Distributions,” Electronic Journal of Statistics, 7, 1747–1782. Granger,C.W.J.(1999): “Outline of Forecast Theory Using Generalized Cost Functions,” Spanish Economic Review, 1, 161–173. Granger, C. W. J., and M. H. Pesaran (2000): “A DecisionTheoretic Approach to Forecast Evaluation,” in Statistics and Finance: An Interface, ed. by W.S. Chan, W. K. Li, and H. Tong. Imperial College Press. Hall, S. G., and J. Mitchell (2007): “Combining Density Forecasts,” International Journal of Forecasting, 23, 1–13. Hansen, L. P. (1982): “Large Sample Properties of Generalized Method of Moments Estimators,” Econometrica, 50(4), 1029–1054. Holzmann, H., and M. Eulert (2011): “The Role of the Information Set for Forecasting — With Applications to Risk Management,” (unpublished). Kallenberg, O. (2002): Foundations of Modern Probability. Springer, 2 edn. Kupiec, P. H. (1995): “Techniques for Verifying the Accuracy of Risk Measurement Models,” Journal of Derivatives, 3(2), 73–84. Lopez, J. A. (1998): “Methods for Evaluating ValueatRisk Estimates,” Economic Policy Review, (October), 119–124. Mincer, J. A., and V. Zarnowitz (1969): “The Evaluation of Economic Forecasts,” in Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, ed. by J. A. Mincer, pp. 3–46. National Bureau of Economic Research. Mitchell, J., and K. F. Wallis (2011): “Evaluating Density Forecasts: Forecast Combinations, Model Mixtures, Calibration and Sharpness,” Journal of Applied Econometrics, 26(6), 1023– 1040. Murphy, A. H., and R. L. Winkler (1987): “A General Framework for Forecast Verification,” Monthly Weather Review, 115, 1330–1338. Pesaran, M. H., and S. Skouras (2002): “DecisionBased Methods for Forecast Evaluation,” in A Companion to Economic Forecasting, ed. by M. P. Clements, and D. F. Hendry, chap. 11, pp. 241–267. Blackwell. RiskMetrics (1996): “RiskMetrics(TM) — Technical Document (4th ed.),” Discussion paper, J. P. Morgan/Reuters‘. Sanders, F. (1963): “On Subjective Probability Forecasting,” Journal of Applied Meteorology, 2, 191–201. Sarno, L., and G. Valente (2004): “Comparing the Accuracy of Density Forecasts from Competing Models,” Journal of Forecasting, 23, 541–557. Shiller, R. J. (1978): “Rational Expectations and the Dynamic Structure of Macroeconomic Models: A Critical Review,” Journal of Monetary Economics, 4(1), 1–44. Tsyplakov, A. (2011): “Evaluating Density Forecasts: A Comment,” MPRA Paper 32728, University Library of Munich, Germany. Wallis, K. F. (2003): “ChiSquared Tests of Interval and Density Forecasts, and the Bank of England’s Fan Charts,” International Journal of Forecasting, 19(2), 165–175. West, K. D. (1996): “Asymptotic Inference about Predictive Ability,” Econometrica, 64, 1067– 1087. White, H. (2000): “A Reality Check for Data Snooping,” Econometrica, 68(5), 1097–1126. 
URI:  https://mpra.ub.unimuenchen.de/id/eprint/55017 
Available Versions of this Item
 Theoretical guidelines for a partially informed forecast examiner. (deposited 03. Apr 2014 10:59) [Currently Displayed]