Does BIC Estimate and Forecast Better than AIC?

Medel, Carlos A. and Salgado, Sergio C. (2012): Does BIC Estimate and Forecast Better than AIC?

Abstract

We test two questions: (i) Is the Bayesian Information Criterion (BIC) more parsimonious than Akaike Information Criterion (AIC)?, and (ii) Is BIC better than AIC for forecasting purposes? By using simulated data, we provide statistical inference of both hypotheses individually and then jointly with a multiple hypotheses testing procedure to control better for type-I error. Both testing procedures deliver the same result: The BIC shows an in- and out-of-sample superiority over AIC only in a long-sample context.

• Item Type: MPRA Paper
• Language: English
• Keywords: AIC; BIC; time-series models; overfitting; forecast comparison; joint hypothesis testing
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C51 - Model Construction and Estimation; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C53 - Forecasting and Prediction Methods ; Simulation Methods; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C22 - Time-Series Models ; Dynamic Quantile Regressions ; Dynamic Treatment Effect Models ; Diffusion Processes

References

Akaike, H., 1974, "A New Look at the Statistical Model Identification," IEEE Transactions on Automatic Control 19(6): 716-723.

Bhansali, R.J. and D.Y. Downham, 1977, "Some Properties of the Order of an Autoregressive Model Selected by a Generalization of Akaike's EPF Criterion," Biometrika 64(3): 547-551.

Bickel, P. and P. Zhang, 1992, "Variable Selection in Nonparametric Regression with Categorical Covariates," Journal of the American Statistical Association 87: 90-97.

Bozdogan, H., 1987, "Model Selection and Akaike's Information Criterion (AIC): The General Theory and its Analytical Extensions," Psychometrika 52(3): 345-370.

Burhnam, K.P. and D.R. Anderson, 1998, Model Selection and Inference: A Practical Information Theoretic Approach, Springer, New York.

Canova, F., 2007, Methods for Applied Macroeconomic Research, Princeton University Press, USA.

Chen, L., P. Giannakouros, and Y. Yang, 2007, "Model Combining in Factorial Data Analysis," Journal of Statistical Planning and Inference 137(9): 2920-2934.

Clark, T.E. and M. McCracken, 2001, "Tests of Equal Forecast Accuracy and Encompassing for Nested Models," Journal of Econometrics 105(1): 85-110.

Clark, T.E., 2004, "Can Out-of-Sample Forecast Comparisons Help to Prevent Overfitting?," Journal of Forecasting 23(2): 115-139.

Corradi, V. and W. Distaso, 2011, Multiple Forecast Model Evaluation, in M.P. Clements and D.F. Hendry (eds.), The Oxford Handbook of Economic Forecasting, Oxford University Press, USA.

Diebold, F.X., and R. Mariano, 1995, "Comparing Predictive Accuracy," Journal of Business and Economic Statistics 13(3): 253-263.

Fischer, S., 1981, "Relative Shocks, Relative Price Volatility, and Inflation," Brooking Papers on Economic Activity 2: 381-431.

Geweke, J. and R. Meese, 1981, "Estimating Regression Models of Finite but Unknown Order," International Economic Review 22(1): 55-70.

Gordon, R.J. and S.R. King, 1982, "The Output Cost of Desinflation in Traditional and Vector Autoregressive Models," Brooking Papers on Economic Activity 13(1): 205-244.

Granger, C.W.J., and Y. Jeon, 2004, "Forecasting Performance of Information Criteria with Many Macro Series," Journal of Applied Statistics 31(10): 1227-1240.

Hansen, P.R., 2005, "A Test of Superior Predictive Ability," Journal of Business and Economic Statistics 23: 365-380.

Hansen, P.R., 2009, "In-Sample Fit and Out-of-Sample Fit: Their Joint Distribution and its Implications for Model Selection," preliminary version April 23, 2009, Department of Economics, Stanford University, USA.

Holm, S., 1979, "A Simple Sequentially Rejective Multiple Test Procedure," Scandinavian Journal of Statistics 6: 65-70.

Hurvich, C.M. and C.-L. Tsai, 1993, "A Corrected Akaike Information Criterion for Vector Autoregressive Model Selection," Journal of Time Series Analysis 14: 271-279.

Karagrigoriou, A., K. Mattheou, and I. Vonta, 2011, "On Asymptotic Properties of AIC Variants with Applications," American Open Journal of Statistics 1: 105-109.

Kilian, L., 2001, "Impulse Response Analysis in Vector Autoregressions with Unknown Lag Order," Journal of Forecasting 20(3): 161-179.

Koehler, A.B. and E.S. Murphree, 1988, "A Comparison of the Akaike and Schwarz Criteria for Selecting Model Order," Journal of the Royal Statistical Society, Series C (Applied Statistics) 37(2): 187-195.

Kuha, J., 2004, "AIC and BIC: Comparison of Assumptions and Performance," Sociological Methods and Research 33(2): 188-229.

Lütkepohl, H., 1985, "Comparison of Criteria for Estimating the Order of a Vector Autoregressive Process," Journal of Time Series Analysis 6(1): 35-52.

Markon, K.E. and R.F. Krueger, 2004, "An Empirical Comparison of Information - Theoretic Selection Criteria for Multivariate Behavior Genetic Models," Behavior Genetics 34(6): 593-609.

Miller, P., T.M. Supel, and T.H. Turner, 1980, "Estimating the Effects of the Oil-Price Shock," Quarterly Review, Federal Reserve Bank of Minneapolis.

Newey, W. and K. West, 1987, "A Simple, Positive Semi-definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix," Econometrica 55(3): 703-708.

Nickelsburg, G., 1985, "Small-Sample Properties of Dimensionality Statistics for Fitting VAR Models to Aggregate Economic Data -A Monte Carlo Study," Journal of Econometrics 28(2): 183-192.

Nishii, R., 1984, "Asymptotic Properties of Criteria for Selection of Variables in Multiple Regression," Annals of Statistics 12(2): 758-765.

Pincheira, P., 2011a, A Joint Test of Superior Predictive Ability for Chilean Inflation, (in Spanish) Working Paper 620, Central Bank of Chile.

Pincheira, P., 2011b, A Bunch of Models, a Bunch of Nulls and Inference About Predictive Ability, Working Paper 607, Central Bank of Chile.

Pincheira, P., 2012, "A Joint Test of Superior Predictive Ability for Chilean Inflation," (in Spanish) forthcoming in Journal Economía Chilena (The Chilean Economy).

Pötscher, B.M. and S. Srinivasan, 1991, "A Comparison of Order Estimation Procedures for ARMA Models," Statistica Sinica 4: 29-50.

Raffalovich, L.E., G.D. Deane, D. Armstrong, and H.-S. Tsao, 2008, "Model Selection Procedures in Social Research: Monte-Carlo Simulation Results," Journal of Applied Statistics 35(10): 1094-1114.

Rissasen, J., 1978, "Modeling by Shortest Data Description," Automatica 14(5): 465-471.

Sawa, T., 1978, "Information Criteria for Discriminating Among Alternative Regression Models," Econometrica 46(6): 1273-1282.

Sargent, T. and C. Sims, 1977, "Business Cycle Modeling Without Pretending to Have too Much a priori Economic Theory," Working Paper 55, Federal Reserve Bank of Minneapolis, USA.

Schwarz, G.E., 1978, "Estimating the Dimension of a Model," Annals of Statistics 6(2): 461-464.

Shibata, R., 1976, "Selection of the Order of an Autoregressive Model by Akaike Information Criterion," Biometrika 63(1): 117-126.

Shibata, R., 1980, "Asymptotically Efficient Selection of the Order of the Model for Estimating Parameters of a Linear Process," Annals of Statistics 8(1): 147-164.

Shibata, R., 1981, "An Optimal Selection of Regression Variables," Biometrika 68: 45-54.

Shittu, O.I. and M.J. Asemota, 2009, "Comparison of Criteria for Estimating the Order of Autoregressive Process: A Monte Carlo Approach," European Journal of Scientific Research 30(3): 409-416.

Sims, C., 1980, "Macroeconomics and Reality," Econometrica 48(1): 1-48.

Sullivan, R., A. Timmermann, and H. White, 1999, "Data-Snooping, Technical Trading Rule Performance, and the Bootstrap," Journal of Finance 54(5): 1647-1691.

Stone, M., 1979, "Comments on Model Selection Criteria of Akaike and Schwarz," Journal of the Royal Statistical Society, Series B (Methodological) 41(2): 276-278.

Yi, G. and G. Judge, 1988, "Statistical Model Selection Criteria," Economic Letters 28(1): 47-51.

Wasserman, L., 2000, "Bayesian Model Selection and Model Averaging," Journal of Mathematical Psychology 44: 92-107.

Weakliem, L.D., 2004, "Introduction to the Special Issue on Model Selection," Sociological Methods and Research 33: 167-186.

West, K., 1996, "Asymptotic Inference about Predictive Ability," Econometrica 64(5): 1067-1084.

White, H., 2000, "A Reality Check for Data Snooping," Econometrica 68: 1097-1126.

Wolak, F.A., 1987, "An Exact Test for Multiple Inequality and Equality Constraints in the Linear Regression Model," Journal of the American Statistical Association 82: 782-793.

Wolak, F.A., 1989, "Testing Inequality Constraints in Linear Econometric Models," Journal of Econometrics 31: 205-235.

Xu, C.J. and I. McLeod, 2012, "Further Asymptotic Properties of the Generalized Information Criteria," Electronic Journal of Statistics 6: 656-663.

Yang, Y., 2003, "Regression with Multiple Candidate Models: Selecting or Mixing?," Statistica Sinica 13: 783-809.

Zucchini W., 2000, "An Introduction to Model Selection," Journal of Mathematical Psychology 44: 41-46.