Illusions in Regression Analysis

Armstrong, J. Scott (2011): Illusions in Regression Analysis. Published in: International Journal of Forecasting (2012)

Abstract

Soyer and Hogarth’s article, “The Illusion of Predictability,” shows that diagnostic statistics that are commonly provided with regression analysis lead to confusion, reduced accuracy, and overconfidence. Even highly competent researchers are subject to these problems. This overview examines the Soyer-Hogarth findings in light of prior research on illusions associated with regression analysis. It also summarizes solutions that have been proposed over the past century. These solutions would enhance the value of regression analysis.

• Item Type: MPRA Paper
• Language: English
• Keywords: a priori analysis, decision-making, ex ante testing, forecasting, non-experimental data, statistical significance, uncertainty
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General

References

Allen, P. G. & Fildes, R. (2001). Econometric forecasting. In J. S. Armstrong. Principles of Forecasting, Boston: Kluwer Academic Publishers,

Armstrong, J. S. (2011). Evidence-based advertising: An application to persuasion. International Journal of Advertising, 30 (5), 743-767.

Armstrong, J. S. (2001). Judgmental bootstrapping: Inferring experts' rules for forecasting.” In J. S. Armstrong. Principles of Forecasting. Boston: Kluwer Academic Publishers.

Armstrong, J. S. (1985). Long-Range Forecasting. New York: John Wiley.

Armstrong, J. S. (1970). How to avoid exploratory research. Journal of Advertising Research, 10, No. 4, 27-30.

Armstrong, J. S. (1968a). Long-range Forecasting for a Consumer Durable in an International Market. PhD Thesis: MIT. (URL to be added)

Armstrong, J. S. (1968b). Long-range forecasting for international markets: The use of causal models. In R. L. King, Marketing and the New Science of Planning. Chicago: American Marketing Association. Available at http://goo.gl/oKYUH.

Armstrong, J. S. & Collopy, F. (1992). Error measures for generalizing about forecasting methods: Empirical comparisons, International Journal of Forecasting, 8, 69-80.

Armstrong, J. S. & Graefe, A. (2011). Predicting elections from biographical information about candidates: A test of the index method. Journal of Business Research, 64, 699-706.

Armstrong, J. S. & Patnaik, S. (2009). Using Quasi-experimental data to develop principles for persuasive advertising. Journal of Advertising Research, 49, No. 2, 170-175.

Christ, C. F. (1960). Simultaneous equation estimation: Any verdict yet? Econometrica, 28, 835-845.

Cumming, G. (2012). Understanding the New Statistics: Effect sizes. Confidence Intervals and Meta-Analysis. New York: Routledge.

Dana, J. & Dawes, R. M. (2004). The superiority of simple alternatives to regression for social science predictions. Journal of Educational and Behavioral Statistics, 29 (3), 317-331.

Einhorn, H. J. (1972). Alchemy in the behavioral sciences. Public Opinion Quarterly, 36, 367-378.

Einhorn, H. J. & R. Hogarth (1975). Unit weighting schemes for decision making. Organizational Behavior and Human Performance, 13, 171-192.

Fildes, R. & Makridakis, S. (1995). The impact of empirical accuracy papers on time series analysis and forecasting. International Statistical Review, 63, 289-308.

Freedman, D. A. (1991). Statistical models and shoe leather. Sociological Methodology, 21, 291-313.

Friedman, M.A. & Schwartz, A. J. (1991). Alternative approaches to analyzing economic data. American Economic Review, 81, Appendix 48-49.

Goldstein, D. G. & Gigerenzer, G. (2009). Fast and frugal forecasting. International Journal of Forecasting, 25, 760-772.

Graefe, A., Armstrong, J.S., Cuzán, A. G. & Jones, R.J., Jr. (2011). Combining forecasts: An application to election forecasts, Working Paper.

Grove, W.M. & Meehl, P.E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: the clinical - statistical controversy. Psychology, Public Policy, and Law, 2, 293-323.

Hauer, E. (2004). The harm done by tests of significance. Accident Analysis and Prevention, 36, 495-500.

Hogarth, R. M. (2012). When simple is hard to accept. In P. M. Todd & Gigerenzer, G. (Eds.), Ecological rationality: Intelligence in the world (in press). Oxford: Oxford University Press.

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, 696-701.

Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640-648.

Karni, E. & Shapiro, B. K. (1980). Tales of horror from ivory towers. Journal of Political Economy. 88, No. 1, 210-212.

Kennedy, P. (2002). Sinning in the basement: What are the rules? The ten commandments of applied econometrics. Journal of Economic Surveys, 16, 569-589.

Meehl, P.E. (1978), Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.

Ord, K. (2012). The Illusion of predictability: A call to action. International Journal of Forecasting, 28 xxx-xxx.

Pant, P. N. & Starbuck, W. H. (1990). Innocents in the forest: Forecasting and research methods. Journal of Management, 16, 433-446

Schmidt, F. L. (1971). The relative efficiency of regression and simple unit predictor weights in applied differential psychology. Educational and Psychological Measurement, 31, 699-714.

Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.

Soyer, E. & Hogarth, R. (2012). Illusion of predictability: How regressions statistics mislead experts. International Journal of Forecasting, 28.

Sparks, J. (1844). The Works of Benjamin Franklin. Boston: Charles Tappan Publisher.

Wold, H. & Jureen, L. (1953). Demand Analysis. New York: John Wiley.

Zellner, A. (2001). Keep it sophisticatedly simple. In Keuzenkamp, H. & McAleer, M. Eds. Simplicity, Inference, and Modelling: Keeping it Sophisticatedly Simple. Cambridge University Press, Cambridge.

Ziliak, S. T. (2012). Visualizing uncertainty: On Soyer's and Hogarth's “The illusion of predictability: How regression statistics mislead experts.” International Journal of Forecasting, 28.

Ziliak, S. T. & McCloskey, D. N. (2008). The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. Ann Arbor: The University of Michigan Press. November 20, 2011 (R47).