Probabilidad Clásica de Sobreajuste con Criterios de Información: Estimaciones con Series Macroeconómicas Chilenas

Medel, Carlos A. (2014): Probabilidad Clásica de Sobreajuste con Criterios de Información: Estimaciones con Series Macroeconómicas Chilenas.

Abstract

This paper provides, via Monte Carlo simulations, estimates of the classical probability of overfitting under an autoregressive environment (AR), using the information criteria (IC) of Akaike, Schwarz and Hannan-Quinn (AIC, BIC and HQ), calibrated with Chilean data of total inflation, core inflation, Imacec, and monthly return of the nominal exchange rate Chilean peso-American dollar. This probability corresponds to the number of times when a candidate model has a strictly greater number of coefficients than the true model, divided by the total number of searches. The results indicate that the increased risk of overfitting is obtained with the AIC, followed by HQ and finally the BIC. The highest probability of overfitting is achieved with the AIC, reaching 32 and 30% with the exchange rate and Imacec, respectively, followed by 25 and 22% for total and core inflation. Considering the three IC, it is more likely to obtain an overfitted model by just one coefficient. Also, it is more likely that the overfitting does not exceed 10 coefficients. These results are important as quantifying the extent to which these variables are subject to overfitting risk when represented by AR models. The potential problems carried by overfitting includes: (i) the spurious regression, (ii) distort the estimation of impulse response function, and (iii) affect the predictive accuracy of the variable of interest. The latter problem is analyzed in detail.

• Item Type: MPRA Paper
• English Title: Classical Probability of Overfitting with Information Criteria: Estimations with Chilean Macroeconomic Series
• Language: Spanish
• Keywords: Nonparametric modelling; information criteria; overfitting; out-of-sample analysis
• Subjects: C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C22 - Time-Series Models ; Dynamic Quantile Regressions ; Dynamic Treatment Effect Models ; Diffusion Processes; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C51 - Model Construction and Estimation; E - Macroeconomics and Monetary Economics > E2 - Consumption, Saving, Production, Investment, Labor Markets, and Informal Economy > E23 - Production; E - Macroeconomics and Monetary Economics > E3 - Prices, Business Fluctuations, and Cycles > E31 - Price Level ; Inflation ; Deflation; E - Macroeconomics and Monetary Economics > E3 - Prices, Business Fluctuations, and Cycles > E37 - Forecasting and Simulation: Models and Applications

References

Akaike, H., 1973, Information Theory and an Extension of the Maximum Likelihood Principle, en Petrov, B.N. y F. Csaki, editores, Proceedings of the 2n} Symposium on Information Theory, Akademia Kiado, Budapest, Hungría.

Akaike, H., 1974, "A New Look at the Statistical Model Identification," IEEE Transactions on Automatic Control 19(6): 716-723.

Box, G.E.P. y G.M. Jenkins, 1970, Time Series Analysis: Forecasting and Control, Holden-Day, San Francisco, EEUU.

Canova, F., 2007, Methods for Applied Macroeconomic Research, Princeton University Press, EEUU.

Cavanaugh, J.E., 1997, "Unifying the Derivations for the Akaike and Corrected Akaike Information Criteria," Statistics and Probability Letters 33(2): 201-208.

Clark, T.E., 2004, "Can Out-of-Sample Forecast Comparisons Help Prevent Overfitting?," Journal of Forecasting 23(2): 115-139.

Elliott, G. y A. Timmermann, 2008, "Economic Forecasting," Journal of Economic Literature 46(1): 3-56.

Faust, J. y J. Wright, 2013, Forecasting Inflation, en Elliot, G. y A. Timmermann, editores, Handbook of Economic Forecasting, volume 2, Elsevier, Holanda.

Granger, C.W.J. y Y. Jeon, 2004, "Forecasting Performance of Information Criteria with Many Macro Series," Journal of Applied Statistics 31(10): 1227-1240.

Granger, C.W.J. y Y. Jeon, 2006, "Dynamics of Model Overfitting Measured in Terms of Autoregressive Roots," Journal of Time Series Analysis 27(3): 347-365.

Granger, C.W.J. y P. Newbold, 1974, "Spurious Regressions in Econometrics," Journal of Econometrics 2(1): 111-120.

Hansen, P.R., 2009, "In-Sample Fit and Out-of-Sample Fit: Their Joint Distribution and its Implications for Model Selection," versión preliminar del 23 de Abril de 2009, Departmento de Economía, Stanford University, EEUU.

Hannan, E.J. y B.G. Quinn, 1979, "The Determination of the Order of an Autoregression," Journal of the Royal Statistical Society B 41: 190-195.

Hawkins, D.M., 2004, "The Problem of Overfitting," Journal of Chemical Information and Computer Sciences 44: 1-12.

Hoover, K.D. y S.J. Pérez, 1999, "Data Mining Reconsidered: Encompassing and the General-to-Specific Approach to Specification Search," The Econometrics Journal 2(2): 167-191.

Hurvich, C.M. y C.-L. Tsai, 1993, "A Corrected Akaike Information Criterion for Vector Autoregressive Model Selection," Journal of Time Series Analysis 14: 271-279.

Kilian, L., 2001, "Impulse Response Analysis in Vector Autoregressions with Unknown Lag Order," Journal of Forecasting 20(3): 161-179.

Koehler, A.B. y E.S. Murphree, 1988, "A Comparison of the Akaike and Schwarz Criteria for Selecting Model Order," Journal of the Royal Statistical Society, Series C (Applied Statistics) 37(2): 187-195.

Kullback, S. y R.A. Leibler, 1951, "On Information and Sufficiency," Annals of Mathematical Statistics 22: 79-86.

Kullback, S., 1968, Information Theory and Statistics, Dover, New York, EEUU.

Lütkepohl, H., 1985, "Comparison of Criteria for Estimating the Order of a Vector Autoregressive Process," Journal of Time Series Analysis 6(1): 35-52.

McQuarrie, A.D., 1999, "A Small-Sample Correction for the Schwarz SIC Model Selection Criterion," Statistics and Probability Letters 44(1): 79-86.

Medel, C.A., 2011, "¿Akaike o Schwarz? ¿Cuál utilizar para predecir el PIB chileno?," Monetaria XXXIV(4): 591-615.

Medel, C.A., 2013, "How Informative are In-sample Information Criteria to Forecasting? The Case of Chilean GDP," Latin American Journal of Economics 50(1): 133-161.

Medel, C.A. y S. Salgado, 2013, "Does the BIC Estimate and Forecast Better than the AIC?," Economic Analysis Review 28(1): 47-64.

Nishii, R., 1984, "Asymptotic Properties of Criteria for Selection of Variables in Multiple Regression," Annals of Statistics 12(2): 758-765.

Schwarz, G.E., 1978, "Estimating the Dimension of a Model," Annals of Statistics 6(2): 461-464.

Seghouane, A.-K., 2006, "A Note on Overfitting Properties of KIC and KIC_{c}," Signal Processing 86(10): 3055-3060.

Shibata, R., 1976, "Selection of the Order of an Autoregressive Model by Akaike Information Criterion," Biometrika 63(1): 117-126.

Stock, J.H. y M.W. Watson, 2007, A Comparison of Linear and Nonlinear Univariate Models for Forecasting Macroeconomic Time Series, en Engle, R.E. y H. White, editores, Cointegration, Causality, and Forecasting: A Fetstschrift in Honour of Clive W.J. Granger, Oxford University Press.

Stock, J.H. y M.W. Watson, 2009, Phillips Curve Inflation Forecasts, en Fuhrer, J., Y. Kodrzycki, J.S. Little y G. Olivei, editores, Understanding Inflation and the Implications for Monetary Policy, A Phillips Curve Restrosprective, MIT Press, EEUU.

Xu, C.J. y I. McLeod, 2012, "Further Asymptotic Properties of the Generalized Information Criteria," Electronic Journal of Statistics 6: 656-663.