Golden Rule of Forecasting: Be conservative

Armstrong, J. Scott and Green, Kesten C. and Graefe, Andreas (2014): Golden Rule of Forecasting: Be conservative.

Abstract

This paper proposes a unifying theory of forecasting in the form of a Golden Rule of Forecasting. The Golden Rule is to be conservative. A conservative forecast is consistent with cumulative knowledge about the present and the past. To be conservative, forecasters must seek all knowledge relevant to the problem, and use methods that have been validated for the situation. A checklist of 28 guidelines is provided to implement the Golden Rule. This article’s review of research found 150 experimental comparisons; all supported the guidelines. The average error reduction from following a single guideline (compared to common practice) was 28 percent. The Golden Rule Checklist helps forecasters to forecast more accurately, especially when the situation is uncertain and complex, and when bias is likely. Non-experts who know the Golden Rule can identify dubious forecasts quickly and inexpensively. To date, ignorance of research findings, bias, sophisticated statistical procedures, and the proliferation of big data have led forecasters to violate the Golden Rule. As a result, despite major advances in forecasting methods, evidence that forecasting practice has improved over the past half-century is lacking.

• Item Type: MPRA Paper
• Language: English
• Keywords: accuracy, analytics, bias, big data, causal forces, causal models, combining, complexity, contrary series, damped trends, decision-making, decomposition, Delphi, ethics, extrapolation, inconsistent trends, index method, judgmental bootstrapping, judgmental forecasting, nowcasting, regression, risk, shrinkage, simplicity, stepwise regression, structured analogies
• Subjects: C - Mathematical and Quantitative Methods > C0 - General > C01 - Econometrics; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C15 - Statistical Simulation Methods: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables; C - Mathematical and Quantitative Methods > C3 - Multiple or Simultaneous Equation Models ; Multiple Variables; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling; C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology ; Computer Programs; C - Mathematical and Quantitative Methods > C9 - Design of Experiments; K - Law and Economics > K2 - Regulation and Business Law

References

Allen, P. G. (1994). Economic forecasting in agriculture. International Journal of Forecasting, 10(1), 81–135.

Arkes, H. R., Shaffer, V. A., & Dawes, R. M. (2006). Comparing holistic and disaggregated ratings in the evaluation of scientific presentations. Journal of Behavioral Decision Making, 19(5), 429–439.

Armstrong, J. S. (1970). An application of econometric models to international marketing. Journal of Marketing Research, 7(2), 190–198.

Armstrong, J. S. (1978), Forecasting with econometric methods, Journal of Business, 51 (4), 549–564.

Armstrong, J. S. (1980). The Seer-Sucker Theory: The Value of Experts in Forecasting. Technology Review, 83 (June/July), 18–24.

Armstrong, J. S. (1985). Long-range Forecasting: From Crystal Ball to Computer. New York: Wiley.

Armstrong, J. S. (2001a). Combining forecasts. In J. S. Armstrong (Ed.), Principles of Forecasting: A Handbook for Researchers and Practitioners (pp. 417–439). New York: Springer.

Armstrong, J. S. (2001b). Judgmental bootstrapping: Inferring experts' rules for forecasting. In J. S. Armstrong (Ed.), Principles of Forecasting: A Handbook for Researchers and Practitioners (pp. 171–192). New York: Springer.

Armstrong, J. S. (2001c). Principles of Forecasting: A Handbook for Researchers and Practitioners. New York: Springer.

Armstrong, J. S. (2006a). Findings from evidence-based forecasting: Methods for reducing forecast error. International Journal of Forecasting, 22(3), 583–598.

Armstrong, J. S. (2006b). How to make better forecasts and decisions: Avoid face-to-face meetings. Foresight: The International Journal of Applied Forecasting, 5(2006), 3–8.

Armstrong, J. S. (2010). Persuasive Advertising. New York: Palgrave MacMillan.

Armstrong, J. S. (2012). Illusions in regression analysis. International Journal of Forecasting, 28(3), 689–694.

Armstrong, J. S., Adya, M., & Collopy, F. (2001). Rule-based forecasting: Using judgment in time-series extrapolation. In J. S. Armstrong (Ed.), Principles of Forecasting: A Handbook for Researchers and Practitioners (pp. 259–282). New York: Springer.

Armstrong, J. S., & Andress, J. G. (1970). Exploratory Analysis of Marketing Data: Trees vs. Regression, Journal of Marketing Research, 7, 487–492.

Armstrong, J. S., & Collopy, F. (1992). Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons. International Journal of Forecasting, 8, 69–80.

Armstrong, J. S., & Collopy, F. (1993). Causal forces: Structuring knowledge for time-series extrapolation. Journal of Forecasting, 12(2), 103–115.

Armstrong, J. S. & Collopy, F. (1998). Integration of statistical methods and judgment for time series forecasting: Principles from empirical research. In G. Wright & P. Goodwin (Eds.), Forecasting with Judgment (pp.263–393). Chichester: Wiley.

Armstrong, J. S., Collopy, F., & Yokum, J. T. (2005). Decomposition by causal forces: a procedure for forecasting complex time series. International Journal of Forecasting, 21(1), 25–36.

Armstrong, J. S., & Graefe, A. (2011). Predicting elections from biographical information about candidates: A test of the index method. Journal of Business Research, 64(7), 699–706.

Armstrong, J. S. & Green, K. C. (2013). Effects of corporate social responsibility and irresponsibility policies: Conclusions from evidence-based research. Journal of Business Research, 66, 1922–1927.

Armstrong, J. S., Green, K. C., & Soon, W. (2008). Polar bear population forecasts: A public-policy forecasting audit. Interfaces, 38(5), 382–405.

Armstrong, J. S., Du, R., Green, K. C., Graefe, A., & House, A. (2014). Predictive validity of evidence-based advertising principles. Annual Meeting of the International Communication Association, May 2014, Seattle. Available at https://marketing.wharton.upenn.edu/files/?whdmsaction=public:main.file&fileID=6794

Ascher, W. (1978). Forecasting: An Appraisal for Policy-makers and Planners. Baltimore: The Johns Hopkins University Press.

Booth, H. (2006). Demographic forecasting: 1980 to 2005 in review. International Journal of Forecasting 22 (3), 547-581.

Bunn, D. W., & Vassilopoulos, A. I. (1999). Comparison of seasonal estimation methods in multi-item short-term forecasting. International Journal of Forecasting, 15(4), 431–443.

Carson, R. T., Cenesizoglu, T., & Parker, R. (2011). Forecasting (aggregate) demand for US commercial air travel. International Journal of Forecasting, 27, 923–94.

Chamberlin, T. C. (1890, 1965). The method of multiple working hypotheses. Science, 148, 754–759. (Reprint of an 1890 paper).

Chen, H. & Boylan, J. E. (2008). Empirical evidence on individual, group and shrinkage indices. International Journal of Forecasting, 24, 525–543.

Clemen, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5(4), 559–583.

Collopy, F., & Armstrong, J. S. (1992a). Expert opinions about extrapolation and the mystery of the overlooked discontinuities. International Journal of Forecasting, 8(4), 575–582.

Collopy, F., & Armstrong, J. S. (1992b). Rule-based forecasting: Development and validation of an expert systems approach to combining time series extrapolations. Management Science, 38(10), 1394–1414.

Dangerfield, B. J., & Morris, J. S. (1992). Top-down or bottom-up: Aggregate versus disaggregate extrapolations. International Journal of Forecasting, 8(2), 233–241.

Devarajan, S. (2013). Africa’s statistical tragedy. The Review of Income and Wealth, 59, Special Issue, S9–S15.

Einhorn, H. J. (1972). Alchemy in the behavioral sciences. Public Opinion Quarterly, 36(3), 367–378.

Fildes, R., & Goodwin, P. (2007). Against your better judgment? How organizations can improve their use of management judgment in forecasting. Interfaces, 37(6), 570–576.

Fildes, R., Goodwin, P., Lawrence, M., & Nikolopoulos, K. (2009). Effective forecasting and judgmental adjustments: An empirical evaluation and strategies for improvement in supply-chain planning. International Journal of Forecasting, 25(1), 3–23.

Fildes, R., & Hastings, R. (1994). The organization and improvement of market forecasting. The Journal of the Operational Research Society, 45(1), 1–16.

Fildes, R., & Makridakis, S. (1995). The impact of empirical accuracy studies on time series analysis and forecasting. International Statistical Review / Revue Internationale de Statistique, 63(3), 289–308.

Flores, B. E, & Whybark, C. D. (1986). A comparison of focus forecasting with averaging and exponential smoothing. Production and Inventory Management, 27(3), 96–103.

Flyvbjerg, B. (2013). Quality control and due diligence in project management: Getting decisions right by taking the outside view. International Journal of Project Management, 31(5), 760–774.

Franses, P. H. & Legerstee, R. (2010). Do experts’ adjustments on model-based SKU-level forecasts improve forecast quality? Journal of Forecasting, 29(3), 331–340.

Freedman, D. A. (1991). Statistical models and shoe leather. Sociological Methodology, 21(1), 201–313.

Gardner, E. S., Jr. (1984). The strange case of the lagging forecasts. Interfaces. 14(3), 47–50.

Gardner, E. S., Jr. (1985). Further notes on lagging forecasts, Interfaces. 15(5), 63.

Gardner, E. S. Jr. (2006). Exponential smoothing: The state of the art—Part II. International Journal of Forecasting, 22, 637–666.

Gardner, E. S. Jr. & Anderson E. A. (1997). Focus forecasting reconsidered, International Journal of Forecasting, 13(4), 501–508.

Gardner, E. S. Jr,. Anderson-Fletcher, E. A., & Wickes, A. M. (2001). Further results on focus forecasting vs. exponential smoothing. International Journal of Forecasting, 17(2), 287–293.

Gawande, A. (2010). The Checklist Manifesto: How to Get Things Right. New York: Metropolitan Books.

Goodwin, P. (this issue). When simple alternatives to Bayes formula work well: Reducing the cognitive load when updating probability forecasts. Journal of Business Research, XXXX.

Goodwin, P. (2000). Improving the voluntary integration of statistical forecasts and judgment. International Journal of Forecasting, 16(1), 85–99.

Goodwin, P., & Fildes, R. (1999). Judgmental forecasts of time series affected by special events: Does providing a statistical forecast improve accuracy? Journal of Behavioral Decision Making, 12(1), 37–53.

Goodwin, P. & Meeran, S. (2012) Robust testing of the utility-based high-technology product sales forecasting methods proposed by Decker and Gnibba-Yukawa (2010). Journal of Product Innovation Management, 29(S1), 211–218.

Gorr, W., Olligschlaeger, A., & Thompson, Y. (2003). Short-term forecasting of crime. International Journal of Forecasting, 19(4), 579–594.

Graefe, A. (this issue). Improving forecasts using equally weighted predictors. Journal of Business Research, XXXX.

Graefe, A., & Armstrong, J. S. (2011). Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task. International Journal of Forecasting, 27(1), 183–195.

Graefe, A., & Armstrong, J. S. (2013). Forecasting elections from voters' perceptions of candidates' ability to handle issues. Journal of Behavioral Decision Making, 26(3), 295–303.

Graefe, A., Küchenhoff, H., Stierle, V. & Riedl, B. (2014). Conditions of Ensemble Bayesian Model Averaging for political forecasting, Working paper, Available at: http://ssrn.com/abstract=2266307.

Graefe, A., Armstrong, J. S., Jones Jr., R. J., & Cuzán, A. G. (2014). Combining forecasts: An application to elections. International Journal of Forecasting, 30(1), 43–54.

Green, K. C., & Armstrong, J. S. (2007a). Global warming: Forecasts by scientists versus scientific forecasts. Energy & Environment, 18(7-8), 997–1021.

Green, K. C., & Armstrong, J. S. (2007b). Structured analogies for forecasting. International Journal of Forecasting, 23(3), 365–376.

Green, K. C., Armstrong, J. S., & Soon, W. (2009). Validity of climate change forecasting for public policy decision making. International Journal of Forecasting, 25(4), 826–832.

Green, K. C., Soon, W., & Armstrong, J. S. (2014). Evidence-based forecasting for climate change. Working paper.

Harvey, N. (1995). Why are judgments less consistent in lss predictable task situations. Organizational Behavior and Human Decision Processes, 63, 247–263.

Hauser, P. M. (1975). Social Statistics in Use. New York: Russell Sage.

Haynes, A. B., Weiser, T. G., Berry, W. R., Lipsitz, S. R., Breizat, A. H. S., & Dellinger, E. P., in Lapitan, M. C. M. (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine, 360(5), 491–499.

Hoch, S. J. (1985). Counterfactual reasoning and accuracy in predicting personal events. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11(4), 719–731.

Hogarth, R. M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21(1), 40–46.

Idso, C. D., Carter, R. M. & Singer, S. F. (2013). Climate Change Reconsidered II: Physical Science. Chicago, Il: The Heartland Institute.

Jørgensen, M. (2004). Top-down and bottom-up expert estimation of software development effort. Information and Software Technology, 46(1), 3–16.

Kabat, G. C. (2008). Hyping Health Risks. New York: Columbia University Press.

Kinney, W. R., Jr. (1971). Predicting earnings: Entity versus subentity data. Journal of Accounting Research, 9, 127–136.

Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980). Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory, 6(2), 107–118.

Larrick, R. P., & Soll, J. B. (2006). Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science, 52(1), 111–127.

Legerstee, R. & Franses, P. H. (2014), Do experts’ SKU forecasts improve after feedback? Journal of Forecasting, 33, 66-79.

MacGregor, D. (2001). Decomposition for judgmental forecasting and estimation. In J. S. Armstrong (Ed.), Principles of Forecasting: A Handbook for Researchers and Practitioners (pp. 107–123). New York: Springer.

Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J. Parzen, E., & Winkler, R. (1982). The Accuracy of Extrapolation (Time Series) Methods: Results of a Forecasting Competition. Journal of Forecasting, 1(2), 111–153.

Makridakis, S., Chatfield, C., Hibon, M., Lawrence, M., Mills, T., Ord, K., & Simmons, L. F. (1993). The M2-competition: A real-time judgmentally based forecasting study. International Journal of Forecasting, 9(1), 5–22.

Makridakis, S., & Hibon, M. (2000). The M3-Competition: results, conclusions and implications. International Journal of Forecasting, 16(4), 451–476.

Malkiel. B. G. (2012). A Random Walk Down Wall Street: The Time-Tested Strategy for Successful Investing (Tenth Edition). New York: W.W. Norton.

Mancuso, A. C. B., & Werner, L. (2013). Review of combining forecasts approaches. Independent Journal of Management & Production, 4(1), 248–277.

McCarthy, T. M., Davis, D. F., Golicic, S. L., & Mentzer, J. T. (2006). The evolutions of sales forecasting management: A 20-year longitudinal study of forecasting practices. Journal of Forecasting, 25, 303-324.

Meehl, P. E. (1954). Clinical versus statistical prediction. Minneapolis: University of Minnesota Press.

Miller, D. M., & Williams, D. (2004). Damping seasonal factors: Shrinkage estimators for the X-12-ARIMA program. International Journal of Forecasting. 20(4), 529-549. (Published with commentary, pp 551–568).

Mollick, E. (2006). Establishing Moore’s Law. IEEE Annals of the History of Computing, 28, 62-75.

Namboodiri, N.K., & Lalu, N.M. (1971). The average of several simple regression estimates as an alternative to the multiple regression estimate in postcensal and intercensal population estimation: A case study. Rural Sociology, 36, 187–194.

Nikolopoulos, K., Litsa, A., Petropoulos, F., Bougioukosa, V., & Khammash, M. (2014). Relative performance of methods for forecasting special events. Journal of Business Research, this issue, XXXX.

Peacock, E., Taylor, M. K., Laake, J., Stirling, I. (2013). Population ecology of polar bears in Davis Strait, Canada and Greenland. The Journal of Wildlife Management, 77, 463–476.

Prasad, V., Vandross, A., Toomey, C., Cheung, M., Rho, J., Quinn, S., Chako, S.J., Borkar, D., Gall, V., Selvaraj, S., Ho, N., & Cifu, A. (2013). A decade of reversal: An analysis of 146 contradicted medical practices. MayoClinicProceedings.org, 790–798. Available from http://www.senyt.dk/bilag/artiklenframayoclinicproce.pdf

Pronovost, P., Needham, D., Berenholtz, S., Sinopoli, D., Chu, H., Cosgrove, S., Sexton, B., Hyzy, R., Welsh, R., Roth, G., Bander, J., Kepros, J., & Goeschel, C. (2006). An intervention to decrease catheter-related bloodstream infections in the ICU. New England Journal of Medicine, 355, 2725–2732.

Randall, D. A., Wood, R. A., Bony, S., Colman, R., Fichefet, T., Fyfe, J., Kattsov, V., Pitman, A., Shukla, J., Srinivasan, J., Stouffer, R. J., Sumi, A., & Taylor, K.E. (2007). Climate Models and Their Evaluation. In S. Solomon, D. Qin, M. Manning, Z. Chen, M. Marquis, K. B. Averyt, M. Tignor, & H. L. Miller (Eds.), Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (pp. 589–662). Cambridge, UK and New York, USA: Cambridge University Press.

Rowe, G., & Wright, G. (2001). Expert opinions in forecasting: The role of the Delphi technique. In J. S. Armstrong (Ed.), Principles of Forecasting: A Handbook for Researchers and Practitioners (pp. 125–144). New York: Springer.

Runkle, D. E. (1998). Revisionist history: how data revisions distort economic policy research. Federal Reserve Bank of Minneapolis Quarterly Review, 22(4), 3–12.

Ryan, P., & Session, J. (2013). Sessions, Ryan Call For Halt On Taxpayer Funding For Risky High-Speed Rail Project. U.S. Senate Budget Committee, Available from http://www.budget.senate.gov/republican/public/index.cfm/2013/3/sessions-ryan-call-for-halt-on-taxpayer-funding-for-risky-high-speed-rail-project.

Sanders, N. R., & Manrodt, K. B. (1994). Forecasting practices in US corporations: Survey results. Interfaces, 24(2), 92–100.

Sanders N. R., & Ritzman L. P. (2001). Judgmental adjustment of statistical forecasts. In J. S. Armstrong (Ed.), Principles of Forecasting: A Handbook for Researchers and Practitioners (pp. 405–416). New York: Springer.

Schnaars, S. P. (1989). Megamistakes: Forecasting and the Myth of Rapid Technological Change. The Free Press: New York.

Schnaars, S. P., & Bavuso, R. J. (1986). Extrapolation models on very short-term forecasts. Journal of Business Research, 14(1), 27–36.

Shu, L. L., Mazar, N., Gino, F., Ariely, D., & Bazerman, M. H. (2012). Signing at the begining makes ethics salient and decreases dishonest self-reports in comparison to signing at the end. Proceedings of the National Academy of Sciences, 109(38), 15197–15200.

Soyer, E., & Hogarth, R. M. (2012). Illusion of predictability: How regression statistics mislead experts. International Journal of Forecasting, 28(3), 695–711.

Sparks, J. (1844). The Works of Benjamin Franklin (Vol. 8). Boston: Charles Tappan Publisher.

Tessier, T. H., & Armstrong, J. S. (2014). Decomposition of Time-Series Forecasts by Current Status and Change: Effects on Accuracy. Working Paper (provide URL)

Tetlock, P. C. (2005). Expert political judgment. Princeton: Princeton University Press.

Tierney, J. (1990). Betting on the planet. New York Times. Available from http://www.nytimes.com/1990/12/02/magazine/betting-on-theplanet.html?pagewanted=all&src=pm.

Vokurka, R. J., Flores, B. E., & Pearce, S. L. (1996). Automatic feature identification and graphical support in rule-based forecasting: A comparison. International Journal of Forecasting, 12, 495–512.

Weimann, G. (1990). The obsession to forecast: Pre-election polls in the Israeli press. Public Opinion Quarterly, 54, 396–408.

Winston, C. (2006). Government Failure versus Market Failure: Microeconomics Policy Research and Government Performance. Washington, D.C.: AEI-Brookings Joint Center for Regulatory Studies. Available from http://www.brookings.edu/press/Books/2006/governmentfailurevsmarketfailure.aspx.

Withycombe, R. (1989). Forecasting with combined seasonal indices. International Journal of Forecasting, 5, 547–552.

Wright, M., & Stern, P. (2014). Forecasting new product trial with analogous series. Journal of Business Research, this issue, XXXX.

Zarnowitz, V. (1967). An appraisal of short-term economic forecasts, NBER Occasional Paper 104, New York: National Bureau of Economic Research.