The Mean Squared Prediction Error Paradox

Pincheira, Pablo and Hardy, Nicolas (2021): The Mean Squared Prediction Error Paradox.

Abstract

In this paper, we show that traditional comparisons of Mean Squared Prediction Error (MSPE) between two competing forecasts may be highly controversial. This is so because when some specific conditions of efficiency are not met, the forecast displaying the lowest MSPE will also display the lowest correlation with the target variable. Given that violations of efficiency are usual in the forecasting literature, this opposite behavior in terms of accuracy and correlation with the target variable may be a fairly common empirical finding that we label here as "the MSPE Paradox." We characterize "Paradox zones" in terms of differences in correlation with the target variable and conduct some simple simulations to show that these zones may be non-empty sets. Finally, we illustrate the relevance of the Paradox with two empirical applications.

• Item Type: MPRA Paper
• Language: English
• Keywords: Mean Squared Prediction Error, Correlation, Forecasting, Time Series, Random Walk.
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C10 - General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C18 - Methodological Issues: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C22 - Time-Series Models ; Dynamic Quantile Regressions ; Dynamic Treatment Effect Models ; Diffusion Processes; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C40 - General; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C53 - Forecasting and Prediction Methods ; Simulation Methods; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C58 - Financial Econometrics; E - Macroeconomics and Monetary Economics > E0 - General; E - Macroeconomics and Monetary Economics > E0 - General > E00 - General; E - Macroeconomics and Monetary Economics > E3 - Prices, Business Fluctuations, and Cycles > E30 - General; E - Macroeconomics and Monetary Economics > E3 - Prices, Business Fluctuations, and Cycles > E31 - Price Level ; Inflation ; Deflation; E - Macroeconomics and Monetary Economics > E3 - Prices, Business Fluctuations, and Cycles > E37 - Forecasting and Simulation: Models and Applications; E - Macroeconomics and Monetary Economics > E4 - Money and Interest Rates > E44 - Financial Markets and the Macroeconomy; E - Macroeconomics and Monetary Economics > E4 - Money and Interest Rates > E47 - Forecasting and Simulation: Models and Applications; E - Macroeconomics and Monetary Economics > E5 - Monetary Policy, Central Banking, and the Supply of Money and Credit > E52 - Monetary Policy; E - Macroeconomics and Monetary Economics > E5 - Monetary Policy, Central Banking, and the Supply of Money and Credit > E58 - Central Banks and Their Policies; F - International Economics > F3 - International Finance > F30 - General; F - International Economics > F3 - International Finance > F31 - Foreign Exchange; F - International Economics > F3 - International Finance > F37 - International Finance Forecasting and Simulation: Models and Applications; G - Financial Economics > G0 - General > G00 - General; G - Financial Economics > G1 - General Financial Markets > G12 - Asset Pricing ; Trading Volume ; Bond Interest Rates; G - Financial Economics > G1 - General Financial Markets > G15 - International Financial Markets; G - Financial Economics > G1 - General Financial Markets > G17 - Financial Forecasting and Simulation; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q0 - General; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q0 - General > Q00 - General; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q0 - General > Q02 - Commodity Markets; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q1 - Agriculture; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q2 - Renewable Resources and Conservation; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q3 - Nonrenewable Resources and Conservation; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q3 - Nonrenewable Resources and Conservation > Q33 - Resource Booms; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q4 - Energy; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q4 - Energy > Q43 - Energy and the Macroeconomy; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q4 - Energy > Q47 - Energy Forecasting

References

1. Ang, A., G. Bekaert and M. Wei. (2007) Do macro variables, asset markets, or surveys forecast inflation better? Journal of Monetary Economics, Volume 54, issue 4, 1163-1212.

2. Bentancor, A. and P. Pincheira. (2010) Predicción de errores de proyección de inflación en Chile. El Trimestre Económico, Vol (0) 305, 129-154. (In Spanish).

3. Chen, Y.-C., Rogoff, K. S., and Rossi, B. (2010). Can Exchange Rates Forecast Commodity Prices ? Quarterly Journal of Economics, 125(August), 1145–1194.

4. Chen, Y.-C., Rogoff, K. S., and Rossi, B. (2011). Predicting Agri-Commodity Prices: An Asset Pricing Approach, World Uncertainty and the Volatility of Commodity Markets, ed. B. Munier, IOS.

5. Clark, T. E., and West, K. D. (2006). Using out-of-sample mean squared prediction errors to test the martingale difference hypothesis. Journal of Econometrics, 135(1-2), 155-186.

6. Clark, T. E., and West, K. D. (2007). Approximately normal tests for equal predictive accuracy in nested models. Journal of Econometrics, 138(1), 291-311.

7. Diebold, F. X., and Mariano, Roberto, S. (1995). Comparing Predictive Accuracy. Journal of Business and Economic Statics, 13(3), 253–263.

8. Engel C. and K. D. West (2005). Exchange Rates and Fundamentals. Journal of Political Economy, June 2005, 113 (3), 485—517.

9. Ferraro, D., Rogoff, K., and Rossi, B. (2015). Can oil prices forecast exchange rates? An empirical analysis of the relationship between commodity prices and exchange rates. Journal of International Money and Finance, 54, 116-141.

10. Giacomini, R., and White, H. (2006). Tests of conditional predictive ability. Econometrica, 74(6), 1545-1578.

11. Ince, O., and Molodtsova, T. (2017). Rationality and forecasting accuracy of exchange rate expectations: Evidence from survey-based forecasts. Journal of International Financial Markets, Institutions and Money, 47, 131-151.

12. Joutz, F. and H.O. Stekler (2000). An Evaluation of the Prediction of the Federal Reserve. International Journal of Forecasting. 16: 17–38.

13. Meese, R, and Rogoff, K. (1983). Empirical exchange rate models of the seventies: Do they fit out of sample? Journal of International Economics, 14, 3–24.

14. Meese, R., and Rogoff, K. (1988). Was it real? The exchange rate‐interest differential relation over the modern floating‐rate period. The Journal of Finance, 43(4), 933-948.

15. Mehra, Y.P. (2002). Survey measures of expected inflation: Revisiting the issues of predictive content and rationality. Federal Reserve Bank of Richmond Economic Quarterly 88, 17-36.

16. Mincer, J. A., and Zarnowitz, V. (1969). The evaluation of economic forecasts. In Economic forecasts and expectations: Analysis of forecasting behavior and performance (pp. 3-46). NBER.

17. Nordhaus, W. (1987). Forecasting efficiency: concepts and applications. The Review of Economics and Statistics, Vol LXIX, N°4. November 1987.

18. Patton, A. and A. Timmermann (2012). Forecast Rationality Tests Based on Multi-Horizon Bounds“. Journal of Business and Economic Statistics 30(1): 1–17.

19. Pincheira, P., and Hardy, N. (2019). Forecasting base metal prices with the Chilean exchange rate. Resources Policy, 62(February), 256–281.

20. Pincheira, P., and Hardy, N. (2021). Forecasting Aluminum Prices with Commodity Currencies. Resources Policy, forthcoming.

21. Pincheira, P. and N. Fernández. (2011). Corrección de algunos errores sistemáticos de predicción de inflación. Monetaria, CEMLA, vol. 0(1), 37-61. (In Spanish).

22. Pincheira, P. (2010). A Real time evaluation of the Central Bank of Chile GDP growth forecasts. Money Affairs, CEMLA, vol. 0(1), 37-73.

23. Pincheira, P. and R. Álvarez. (2009). Evaluation of short run inflation forecasts and forecasters in Chile," Money Affairs, CEMLA, vol. 0(2), 159-180.

24. P. Pincheira. (2012). A joint test of superior predictive ability for Chilean inflation forecasts. Journal Economía Chilena (The Chilean Economy), Central Bank of Chile, vol. 15(3), 04-39, (In Spanish).

25. Souleles N. (2004). Expectations, heterogeneous forecast errors and consumption: Micro evidence from the Michigan consumer sentiment surveys. Journal of Money, Credit and Banking. 36, 39-72.

26. Stock, J. H., and Watson, M. (2003). Forecasting output and inflation: The role of asset prices. Journal of Economic Literature, 41(3), 788-829.

27. Thomas, L.B., 1999. Survey measures of expected U.S. inflation. Journal of Economic Perspectives, 13, 125-144.

28. Timmermann, A. (2008). Elusive return predictability. International Journal of Forecasting, 24(1), 1-18.

29. Welch, I., and A. Goyal (2008). A comprehensive look at the empirical performance of equity premium prediction. The Review of Financial Studies, 21(4), 1455-1508.

30. West, K. D. (1996). Asymptotic Inference about Predictive Ability. Econometrica, 64(5), 1067.

31. West, K. D. (2006). Chapter 3 Forecast Evaluation. Handbook of Economic Forecasting, 1(05), 99–134.