Forecast Evaluation in Macroeconomics and International Finance. Ph.D. thesis, George Washington University, Washington, DC, USA.

Bespalova, Olga (2018): Forecast Evaluation in Macroeconomics and International Finance. Ph.D. thesis, George Washington University, Washington, DC, USA.

Abstract

This dissertation focuses on forecasting rare macroeconomic events, such as GDP declines and currency crises, using non-parametric methods, highlighting the advantages of the Receiver Operating Characteristic (ROC) curves analysis and the value of qualitative information from expert surveys and textual analysis in macroeconomic forecasting. Chapter 1 shows that the qualitative WES survey can produce accurate directional macroeconomic forecasts for the USA by combining the ROC curves analysis with contingency tables, and that the ROC-optimal thresholds yield more accurate predictions than the alternatives in Hutson et al. (2014). Chapter 2 re-examines indicators of currency crises from Kaminsky and Reinhart (1999) and subsequent studies using the ROC curves analysis and shows that the ROC-optimal thresholds issue more accurate signals than the minimum noise-to-signal ratio previously used in the literature. Modified ROC curves display the relationship between the precision of sent signals and the recall of crisis episodes. Forecast combinations using several ad-hoc rules help to improve forecast accuracy. Chapter 3 highlights asymmetric information about the U.S. economy between the Federal Reserve System (FRS) and the Survey of Professional Forecasters (SPF) via textual analysis of the Federal Open Market Committee (FOMC) minutes. It shows that the SPF forecasters pay close attention to the FOMC minutes available to them at the time of the forecast deadline and efficiently use its information in their set. Yet, they could improve their forecasts should the FOMC minutes from the first quarterly meetings become available without a three-week publication lag. However, during their second quarterly meetings, the FOMC policy-makers accounted only for their own earlier assessment of the U.S. macroeconomy – they did not put weight on the SPF forecasts released a few weeks earlier in the same quarter.

• Item Type: MPRA Paper
• Language: English
• Keywords: Forecasting; macroeconomics; international finance; rare events; forecasts accuracy; forecasts evaluation; GDP; currency crises; real exchange rate (RER); trend; foreign reserves, broad money (M2); the ratio of broad money to reserves; exports; World Economic Survey (WES); Federal Reserve System (FRS); Survey of Professional Forecasters (SPF); SPF forecasters; Federal Open Market Committee (FOMC); FOMC minutes; Greenbook (GB); GB forecasts; elicit casts; textual analysis; non-parametric methods; leading indicators; binary indicators; Receiver Operating Characteristic (ROC) curves; ROC-optimal thresholds; ROC curves analysis; in-sample predictive value; out-of-sample predictive value; signal approach; precision; recall; true positive rate; false positive rate; true negative rate; false negative rate; economic surveys; consumer surveys; business tendency surveys; qualitative surveys; quantitative surveys; contingency table; present economic conditions; future economic conditions; regression approach; balance statistics; probability approach; forecast error; consensus scores; forecasting rule; directional accuracy; accuracy ratio; probability of detection; classifier; informative classifier; random guess; J-index; threshold; area under the curve (AUC); future expectations; Early Warning Indicators (EWIs); encompassing tests; rationality tests; orthogonality tests; forecasts unbiasedness; forecasts efficiency.
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C14 - Semiparametric and Nonparametric Methods: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C18 - Methodological Issues: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C23 - Panel Data Models ; Spatio-temporal Models; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C44 - Operations Research ; Statistical Decision Theory; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C53 - Forecasting and Prediction Methods ; Simulation Methods; E - Macroeconomics and Monetary Economics > E1 - General Aggregative Models > E17 - Forecasting and Simulation: Models and Applications; E - Macroeconomics and Monetary Economics > E5 - Monetary Policy, Central Banking, and the Supply of Money and Credit > E52 - Monetary Policy; E - Macroeconomics and Monetary Economics > E5 - Monetary Policy, Central Banking, and the Supply of Money and Credit > E58 - Central Banks and Their Policies; E - Macroeconomics and Monetary Economics > E6 - Macroeconomic Policy, Macroeconomic Aspects of Public Finance, and General Outlook > E65 - Studies of Particular Policy Episodes; E - Macroeconomics and Monetary Economics > E6 - Macroeconomic Policy, Macroeconomic Aspects of Public Finance, and General Outlook > E66 - General Outlook and Conditions; F - International Economics > F3 - International Finance > F31 - Foreign Exchange; F - International Economics > F3 - International Finance > F37 - International Finance Forecasting and Simulation: Models and Applications; F - International Economics > F4 - Macroeconomic Aspects of International Trade and Finance > F43 - Economic Growth of Open Economies; F - International Economics > F4 - Macroeconomic Aspects of International Trade and Finance > F47 - Forecasting and Simulation: Models and Applications

References

Abberger, K., Frey, M., Kesina, M. & Stangl, A. (2009). Indicatoren fur die globale Konjuctur, Ifo Schnelldienst, 62(34/35), 32-41.

Anderson, O. (1952). The business test of the IFO-Institute for economic research. Munich, and its theoretical model. Revue de l'Institut International de Statistique, 20 (1), 1-17.

Ang, A., Bekaert, G., & Wei, M. (2007). Do macro variables, asset markets, or surveys forecast inflation better? Journal of Monetary Economics, 54 (4), 1163-1212.

Balke, N. & Petersen D. (2002). How well does the Beige Book reflect economic activity? Evaluating qualitative information quantitatively. Journal of Money, Credit and Banking, 23(1), 114-136.

Berg, A. & Pattillo, C. (1999a). Predicting currency crises: the indicators approach and an alternative. Journal of International Money and Finance, 18(4), 561-586.

Berg, A & Pattillo, C. (1999b). What caused the Asian crisis: An early warning system approach. Economics Notes, 28(3), 285-334.

Berge, T.J. & Jordà, Ò. (2011). Evaluating the classification of economic activity into recessions and expansions. American Economic Journal: Macroeconomics, 3, 246-277.

Blöchinger, A. & Lieppold, M. (2006). Economic benefits of powerful credit scoring. Journal of Banking and Finance, 30 (3), 851-873.

Bussiere, M. & Fratzscher, M. (2006). Towards a new early warning system of financial crises. Journal of International Money and Finance, 25 (6), 953-973.

Bussiere, M. (2013). Balance of payment crises in emerging markets: how early were the ‘early’ warning signals? Applied economics, 45 (12), 1601-1620.

Burkart, O. and Coudert, V. (2002). Leading indicators of currency crises for emerging countries. Emerging Markets Review, 3 (2), 107-133.

Candelon, B., Dumitrescu, E-I. & Hurlin, C. (2012). How to evaluate an Early Warning System? Toward a unified statistical framework for assessing financial crises forecasting methods. IMF Economic Review, 60 (1), 75-113.

Candelon, B., Dumitrescu, E-I. & Hurlin, C. (2012). Currency crises early warning systems: why they should be dynamic? International Journal of Forecasting, 30 (4), 1016-1029.

Carlson, J.A. & Parkin, M. (1975). Inflation Expectations. Economica, 42 (165), 123-138.

Carnot, N., Koen, V., & Tissot, B. (2005). Economic Forecasting. New York, NY: Palgrave Macmillian.

Chong, Y. & D. Hendry. (1986). Econometric evaluation of linear macro-economic models. Review of Economic Studies, 53 (4), 671-690.

Claveria, O., Pons, E. & Ramos, R. (2007). Business and consumer expectations and macroeconomic forecasts. International Journal of Forecasting, 23 (1), 47-69.

Clements, M. P. (2015). Are professional macroeconomic forecasters able to do better than forecasting trends? Journal of Money, Credit and Banking, 47 (2-3), 349-382.

Croushore, D. (2005). Do consumer confidence indexes help forecast consumer spending in real time? North American Journal of Economics and Finance, 16 (3), 438-450.

Croushore, D. & Stark, T. (2001). A real-time database for macroeconomists. Journal of Econometrics, 105 (1), 111-130.

Cunningham, A. (1997). Quantifying Survey Data. Bank of England Quarterly Bulletin, 37 (3), 292-300.

Danker, D. J. & Luecker, M.M. (2005). Background in FOMC Meeting Minutes. Federal Reserve Bulletin, 91 (2), 175-179.

DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating curves: a nonparametric approach. Biometrics, 44 (3), 837–845.

Dorfman, D.D. & Alf E. (1969). Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals-rating-method data. Journal of Mathematical Psychology, 6 (3), 487-496.

Doswell, D-J., Davies-Jones, R. & Keller, D.L. (1990). On summary measures of skill in rare events forecasting based on contingency tables. Weather and forecasting, 5(4), 576 – 585.

Drehmann, M. & Juselius, M. (2014). Evaluating early warning indicators of banking crises: Satisfying policy requirements. International Journal of Forecasting, 30, p.759-780.

Edison, H.J. (2003). Do indicators of financial crises work? An evaluation of an early warning system. International Journal of Finance and Economics, 8 (1), 11-53.

Ericsson, N. R. (2012). Economics 8378.10 (Economic Forecasting): Course Lecture Notes. Washington, DC, USA [Unpublished manuscript].

Ericsson, N. R. (2016). Eliciting GDP forecasts from the FOMC minutes around the financial crisis. International Journal of Forecasting, 32 (2), 571-583.

Faust, J. & Wright, J.H. (2009). Comparing Greenbook and reduced form forecasts using a large realtime dataset. Journal of Business and Economic Statistics, 27 (4), 468-479.

Fisher, R.A. (1922). On the interpretation of χ2 from contingency tables, and the calculation of P. Journal of Royal Statistical Society, 85 (1), 87-94.

Fisher, R.A. (1950). Statistical Methods for Research Workers (11th ed.). New York, NY: Hafner Publishing Company.

Frost, J., and Saiki, A. (2014). Early Warning for Currency Crises: What is the role of financial openness? Review of International Economics, 22 (4), 722-743.

Gamber, E.N. & J. Smith, J.K. (2009). Are the Fed’s inflation forecasts still superior to the private sector’s? Journal of Macroeconomics, 31(2), 240-251.

Garnitz, J. (2017). Ifo World Economic Survey – Description and Information. Munich, Germany: Ifo Institute. Retrived from https://www.cesifo-group.de/dms/ifodoc/docs/facts/survey/WES/Description_WES_2017.pdf

Goldfarb, R.S., Stekler, H. O. & David, J. (2005). Methodological issues in forecasting: Insights from the egregious business forecast errors of late 1930, Journal of Economic Methodology, 12 (4), 517-542.

Gorr, W.L. & Schneider, M.J. (2013). Large-change forecast accuracy: Reanalysis of M3-competition data using receiver operating characteristic analysis. International Journal of Forecasting, 29 (2), 274-281.

Granger, C.W.J. & Pesaran, M.H. (2000). Economic and statistical measures of forecast accuracy. Journal of Forecasting, 19 (7), 537-560.

Hamella, S. & Haupt, H. (2007). Suitability of WES data for forecasting inflation. In G. Goldrian (Ed.), Handbook of Survey-Based Business Cycle Analysis (99-115). Cheltenham, United Kingdom & Nortmapton, MA: Edward Elgar Publishing Limited.

Haupt, H. & Waller, S. (2000). Economic analysis and short-term forecasting with qualitative data from the Economic Survey International. In K.H. Oppenlander, G. Poser, and B. Schips (Eds.), Use of Survey Data for Industry, Research, and Economic Policy: CIRET Conference (527-547). Avebury, United Kingdom: Aldershot.

Henriksson, R.D. & Merton, R.C. (1981). On market timing and investment performance 2: Statistical procedures for evaluating forecasting skills. Journal of Business, 54 (4), 513-533.

Henzel, S. & Wollmershäuser, T. (2005). Quantifying Inflation Expectations with Carlson-Parkin Method: A survey-based determination of the just noticeable difference. Journal of Business Cycle Measurement and Analysis, 2005 (3), 321-319.

Henzel, S. & Wollmershäuser, T. (2008). The New Keynesian Phillips curve and the role of expectations: Evidence from the CESifo World Economic Survey. Economic Modelling, 25 (5), 811-832.

Hutson, M., Joutz, F. & Stekler, H. (2014) Interpreting and evaluating CESIfo’s World Economic Survey directional forecasts. Economic Modeling, 38, 6-11.

Jordà, Ò. (2014). Assessing the historical role of credit: Business cycles, financial crises and the legacy of Charles S. Pierce. International Journal of Forecasting, 30 (3), 729-740.

Jordà, Ò., Schularick, M. & Taylor, A.M. (2011). Financial crises, credit booms and external liabilities: 140 years of lessons. IMF Economic Review, 59(2), 340-378.

Joutz, F. & Stekler, H. (2000). An evaluation of the predictions of the Federal Reserve. International Journal of Forecasting, 16 (1), 17-38.

Kamin S.B., Schindler, J. & Samuel, S. (2007). The contribution of domestic and external factors to Latin American devaluation crises: An early warning systems approach, International Journal of Finance and Economics, 12 (3), 317-336.

Kaminsky G. L. Lizondo, S. & Reinhart, C. (1998). Leading indicators of currency crises. IMF Staff papers, 45 (1), 1-48.

Kaminsky G. L. & Reinhart C.M. (1999). The twin crises: The causes of banking and balance of payments problems. American Economic Review, 89(3), 473–500.

Kaminsky G.L. (1999). Currency and banking crises: the early warnings of distress. IMF working paper, 178, 1-38.

Kaminsky G. L. (2003). Varieties of currency crises. NBER Working chapter 10193. Cambridge, MA.: National Bureau of Economic Research.

Kaminsky G. L. (2006). Currency crises: Are they all the same? Journal of International Money and Finance, 25 (3), 503-527.

Krzanowsky, W.J. & Hand, D.J. (2009). ROC curves for continuous data. Boca Raton, FL: Chapman & Hall/CRC.

Kudymova, E., Plenk, J. & Wohlrabe, K (2013). Ifo World Economic Survey and the Business Cycle in selected countries. CESIfo Forum, 14 (4), 51-57.

Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143 (1), 29-36.

Hanley, J. A., & McNeil, B. J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148 (3), 839–843.

Lahiri, K., & Monokroussos, G. (2013). Nowcasting US GDP: The role of ISM business surveys. International Journal of Forecasting, 29 (4), 644-658.

Lahiri, K. & Moore, G.H. (1991). Leading economic indicators: New approaches and forecasting records. Cambridge, United Kingdom: Cambridge University Press.

Lahiri, K. & Wang, J.G. (2013). Evaluating probability forecasts for GDP declines using alternative methodologies. International Journal of Forecasting, 29 (1), 175-190.

Lahiri, K. & Zhao, Y. (2015). Quantifying survey expectations: A critical review and generalization of the Carlson-Parkin method, International Journal of Forecasting, 31 (1), 51-62.

Lamont, O.A. (2002). Macroeconomic forecasts and microeconomic forecasters. Journal of Economic Behavior and Organization, 48 (3), 265-280.

Leitch, G. & Tanner, J. E. (1991). Economic Forecast Evaluation: Profits Versus the Conventional Error Measures. The American Economic Review, 81 (3), 580-590.

Ludvigson, S.C. (2004). Consumer confidence and consumer spending. The Journal of Economic Perspectives, 18 (2), 29-50.

Ma, G. & Hall, W.J. (1993). Confidence bands for the receiver operating characteristic curves. Medical decision making, 13 (3), 191-197.

Merton, R.C. (1981). On market timing and investment performance 1: An equilibrium theory of value for market forecasts. Journal of Business, 54 (3), 363-406.

Mincer, J. & Zarnowitz, V. (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance (3-46). New York, National Bureau of Economic Research.

Muth, J.F. (1961). Rational expectations and the theory of price movements. Econometrica, 29 (3), 315-335.

Nardo, M. (2003). The quantification of the qualitative data: a critical assessment. Journal of Economic Surveys, 17 (5), 645-668.

Newey, W.K. & West, K.D. (1987). A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica, 55(3), 703-708. Business Tendency Surveys: A Handbook (2003). Paris, France: OECD Publishing. Retrived from https://www.oecd.org/std/leading-indicators/31837055.pdf.

Öller, L-E. (1990). Forecasting the business cycle using survey data. International Journal of Forecasting, 6 (4), 453-461.

Ottaviani, M. & Sørensen, P.N. (2006). The strategy of professional forecasting. Journal of Financial Economics, 81 (2), 441-466.

Pepe, M. S. (2000). Receiver operating characteristic methodology. Journal of the American Statistical Association, 95 (449), 308-311.

Pepe, M. S. (2003). The Statistical Evaluation of Medical Tests for Classification and Prediction. New York, NY: Oxford University Press.

Pepe, M., Janes, H., & Longton, G. (2009). Estimation and comparison of receiver operating characteristic curves. The Stata Journal, 9 (1), 1-16.

Pesaran, M. H. (1984). Expectations Formations and Macroeconomic Modeling. In P. Malgrange & P-A. Muet (Eds.), Contemporary Macroeconomic Modeling (27-55). Oxford, United Kingdom: Basil Blackwell.

Pesaran, M. H. (1987). The limits to rational expectations. Oxford, United Kingdom: Basil Blackwell.

Peterson, W. W., & Birdsall, T. G. (1953). The Theory of Signal Detectability: Part I. The General Theory. Technical Report 13. Ann Arbor, MI: University of Michigan, Department of Electrical Engineering, Electronic Defense Group.

Romer, C. D. & Romer, D.H. (2000). Federal Reserve information and the behavior of interest rates. American Economic Review, 90(3), 429-457.

Schnader, M.H. & Stekler, H.O. (1990). Evaluating Predictions of Change. The Journal of Business, 63 (1), 99-107.

Seiler, C. (2015). On the robustness of balance statistics with respect to nonresponse. OECD Journal of Business Cycle Measurement and Analysis, 2014 (2), 1-18.

Schularick, M. & Taylor, A. M. (2012). Credit booms gone bust: monetary policy, leverage cycles and financial crises, 1870–2008. American Economic Review, 102(2), 1029–1061.

Sims, C. (2002). The role of models and probabilities in the monetary policy process. Brookings Chapters on Economic Activity, 2002 (2), 1-40.

Soberhart, J. & Keenan, S. (2001). Measuring default accurately. Credit Risk Special Report, Risk, 14, 31-33.

Stangl, A. (2007A). World Economic Survey. In G. Goldrian (Ed.), Handbook of Survey-Based Business Cycle Analysis (57-65). Cheltenham, United Kingdom & Nortmapton, MA: Edward Elgar Publishing Limited.

Stangl, A. (2007B). Der Index für Konjunkturerwartungen des ZEW und des ifo Instituts, WES, sind für Deutschland identisch. Ifo Schnelldienst, 60 (03), 55-56.

Stekler, H.O. (1994). Are economic forecasts valuable? Journal of Forecasting, 13 (6), 495-505.

Stekler, H.O. (1991). Macroeconomic forecast evaluation techniques. International Journal of Forecasting, 7 (3), 375-384.

Stekler, H.O. & Schnader M.H. (1991). Evaluating predictions of change: an application to inflation forecasts. Applied Financial Economics, 1 (3), 135-137.

Stekler, H.O. & Symington, H. (2016). Evaluating qualitative forecasts: The FOMC minutes, 2006-2010. International Journal of Forecasting, 32 (2), 559-570.

Stekler, H.O. & Ye, T. (2017). Evaluating with a leading indicator: an application – the term spread. Empirical Economics, 53 (1), 183-194.

Theil, H. (1952). On the time shape of economic macro variables and the Munich Business test. Review of the International Statistical Institute, 20 (2), 105-121.

Theil, H. (1961). Economic Forecasts and Policy, (2nd ed.). Amsterdam, Netherlands: North Holland Pub. Co.

Trueman, B. (1994). Analyst Forecasts and Herding Behavior. The Review of Financial Studies, 7 (1), 97-124.

Youden, W.J. (1950). Index for rating diagnostic tests. Cancer, 3 (1), 32–35.