An evaluation of simple forecasting model selection rules

Fildes, Robert and Petropoulos, Fotios (2013): An evaluation of simple forecasting model selection rules.

Abstract

A major problem for many organisational forecasters is to choose the appropriate forecasting method for a large number of data series. Model selection aims to identify the best method of forecasting for an individual series within the data set. Various selection rules have been proposed in order to enhance forecasting accuracy. In theory, model selection is appealing, as no single extrapolation method is better than all others for all series in an organizational data set. However, empirical results have demonstrated limited effectiveness of these often complex rules. The current study explores the circumstances under which model selection is beneficial. Three measures are examined for characterising the data series, namely predictability (in terms of the relative performance of the random walk but also a method, theta, that performs well), trend and seasonality in the series. In addition, the attributes of the data set and the methods also affect selection performance, including the size of the pools of methods under consideration, the stability of methods’ performance and the correlation between methods. In order to assess the efficacy of model selection in the cases considered, simple selection rules are proposed, based on within-sample best fit or best forecasting performance for different forecast horizons. Individual (per series) selection is contrasted against the simpler approach (aggregate selection), where one method is applied to all data series. Moreover, simple combination of methods also provides an operational benchmark. The analysis shows that individual selection works best when specific sub-populations of data are considered (trended or seasonal series), but also when methods’ relative performance is stable over time or no method is dominant across the data series.

• Item Type: MPRA Paper
• Language: English
• Keywords: automatic model selection, comparative methods, extrapolative methods, combination, stability
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C22 - Time-Series Models ; Dynamic Quantile Regressions ; Dynamic Treatment Effect Models ; Diffusion Processes

References

Adya, M.; Collopy, F.; Armstrong, J. & Kennedy, M. (2001), 'Automatic identification of time series features for rule-based forecasting', International Journal of Forecasting 17, 143-157.

Armstrong, J. S., ed. (2001), Principles of Forecasting: A Handbook for Researchers and Practitioners, Boston and Dordrecht: Kluwer.

Assimakopoulos, V. & Nikolopoulos, K. (2000), 'The Theta model: a decomposition approach to forecasting', International Journal of Forecasting 16(4), 521 - 530.

Billah, B.; King, M. L.; Snyder, R. D. & Koehler, A. B. (2006), 'Exponential smoothing model selection for forecasting', International Journal of Forecasting 22(2), 239 - 247.

Collopy, F. & Armstrong, J. (1992), 'Rule-based forecasting: development and validation of an expert systems approach to combining time series extrapolations', Management Science 38, 1392-1414.

Crone, S. & Kourentzes, N. (2011), ''Automatic Model Selection of Exponential Smoothing - an empirical evaluation of Trace Errors in forecasting for Logistics''31st Annual International Symposium on Forecasting ISF 2011, June 26-29, 2011, Prague, Czech Republic.'.

Davydenko, A. & Fildes, R. (2013), 'Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts', International Journal of Forecasting(0), - .

Fildes, R. (2001), 'Beyond forecasting competitions', International Journal of Forecasting 17, 556-560.

Fildes, R. (1989), 'Evaluation of aggregate and individual forecast method selection rules', Management Science 39, 1056-1065.

Goodrich, R. L. (2000), 'The Forecast Pro methodology', International Journal of Forecasting 16(4), 533 - 535.

Hyndman, R. J. & Khandakar, Y. (2008), 'Automatic Time Series Forecasting: The forecast Package for R', Journal of Statistical Software 27(3), 1 - 22.

Hyndman, R. J.; Koehler, A. B.; Snyder, R. D. & Grose, S. (2002), 'A state space framework for automatic forecasting using exponential smoothing methods', International Journal of Forecasting 18(3), 439 - 454.

Makridakis, S. & Hibon, M. (2000), 'The M3-Competition: results, conclusions and implications', International Journal of Forecasting 16(4), 451 - 476.

Makridakis, S. & Winkler, R. L. (1989), 'Sampling distributions of post-sample forecasting errors', Applied Statistics-Journal of the Royal Statistical Society Series C 38, 331-342.

Meade, N. (2000), 'Evidence for the selection of forecasting methods', Journal of Forecasting 19, 515-535.

Ord, K. & Fildes, R. (2013), Principles of Business Forecasting, South-Western Cengage Learning.

Pant, P. N. & Starbuck, W. H. (1990), 'Innocents in the Forest - Forecasting and Research Methods', Journal of Management 16, 433-460.

Shah, C. (1997), 'Model selection in univariate time series forecasting using discriminant analysis', International Journal of Forecasting 13, 489-500.