Testing Theories with Qualitative and Quantitative Predictions

Coleman, Stephen (2005): Testing Theories with Qualitative and Quantitative Predictions.

Abstract

Researchers in the social sciences as well as other disciplines rely on statistical models to develop and test theories. The standard approach applies multivariate analysis—usually linear regression—and statistical hypothesis testing to observational data. The researcher may have several independent variables in mind as candidate predictors of the dependent variable; those reaching statistical significance compose the final model. In other situations, a theory is assumed to be correct and regression analysis is used to estimate parameters. Though experiments with random assignment of subjects are recognized as the “gold standard” for research, these are rarely possible in political science. Instead we rely on statistical controls to overcome problems inherent in using nonexperimental data.

This paper suggests that our confidence in using statistical methods to construct theories is misplaced and that theory testing is more productive when we combine definitive theory-generated predictions with statistical methods. I begin with a review of problems in the current approach to statistical analysis, then give several examples of how prediction can be improved.

• Item Type: MPRA Paper
• Language: English
• Keywords: economic theory, theory testing, statistical testing
• Subjects: B - History of Economic Thought, Methodology, and Heterodox Approaches > B4 - Economic Methodology > B41 - Economic Methodology; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C18 - Methodological Issues: General

References

Ahn, Woo-kyong, Charles W. Kalish, Douglas L. Medin, and Susan A. Gelman. 1995. “The role of covariation versus mechanism information in causal attribution.” Cognition 54: 299-352.

Blais, Andre. 2000. To Vote or Not to Vote: The Merits and Limits of Rational Choice Theory. Pittsburgh, PA: University of Pittsburgh Press.

Buehner, Marc J., Patricia W. Cheng, and Deborah Clifford. 2003. “From covariation to causation: a test of the assumption of causal power.” Journal of Experimental Psychology: Learning, Memory and Cognition 29:1119-1140.

Chatfield, C. 1995. “Model uncertainty, data mining and statistical inference.” Journal of the Royal Statistical Society Series A 158: 419-466.

Cheng, Patricia W. 1997. “From covariation to causation: a causal power theory.” Psychological Review 104: 367-405.

Cialdini, Robert B., Raymond R. Reno, and Carl A. Kallgren. 1990. "A focus theory of normative conduct: Recycling the concept of norms to reduce littering in public places." Journal of Personality and Social Psychology 58: 1015-26.

Clauset, Aaron and Maxwell Young. 2005. “Scale invariance in global terrorism.” http://arxiv.org/abs/physics/0502014.

Coleman, Stephen. 2004. "The effect of social conformity on collective voting behavior," Political Analysis 12: 76-96. For ancillary materials see http://polmeth.wustl.edu/polanalysis/ancillary12.html.

----. 2002. "A test for the effect of conformity on crime rates using voter turnout." The Sociological Quarterly 43: 257-276.

----. 1995. "Dynamics in the fragmentation of political party systems," Quality and Quantity 29: 141-155.

----. 1993. "Cycles and chaos in political party voting." Journal of Mathematical Sociology 18 (1993): 47-64.

----. 1985. "The human brain, social conformity, and presidential elections." Journal of Mathematical Sociology 11: 95 130.

----. 1975. Measurement and Analysis of Political Systems: A Science of Social Behavior. New York: John Wiley & Sons.

Coles, Stuart G. 2001. An Introduction to Statistical Modeling of Extreme Values. New York: Springer Verlag.

Dharmapala, Dhammika and Michael McAleer. 1996. “Econometric methodology and the philosophy of science.” Journal of Statistical Planning and Inference 49: 9-37.

Hauser, J. R. and B. Wernerfelt. 1990. "An evaluation cost model of consideration sets." Journal of Consumer Research 16: 393-408.

Heckman, James. 2000. “Causal parameters and policy analysis in economics: a twentieth century retrospective.” Quarterly Journal of Economics 115: 45ff

Ioannidis, John P.A. 2005. “Contradicted and initially stronger effects in highly cited clinical research.” Journal of the American Medical Association 294: 218-228.

Iyengar, Sheena S. and Mark R. Lepper. 1999. "Rethinking the value of choice: A cultural perspective on intrinsic motivation. Journal of Personality and Social Psychology 76: 349-366.

Kirchgaessner, Gebhard and Anne Meyer zu Himmern. 1997. “Expected closeness and turnout: An empirical analysis of German general elections, 1983-1994.” Public Choice 91: 3-25.

Knack, Stephen and Martha E. Kropf. 1998. "For shame! The effect of community cooperative context on the probability of voting." Political Psychology 19: 585-99.

Knack, Stephen. 1992. "Civic norms, social sanctions, and voter turnout." Rationality and Society 4: 133-56.

Lindgren, Bernard W. 1962. Statistical Theory. New York: Macmillan.

Longford, N. T. 2005. “Editorial: Model selection and efficiency—is ‘Which model . . . ?’ is the right question?” Journal of the Royal Statistical Society Series A 168: 469-472.

Mackie, T. T. and R. Rose. 1991. The International Almanac of Electoral History (3rd ed.). Washington, D.C.: Congressional Quarterly.

Malinetskii, G. G. and S. P. Kurdyumov. 2001. “Nonlinear dynamics and the problems of prediction.” Herald of the Russian Academy of Sciences 71(2): 94-113. Retrieved on 7 April 2005 from http://www.keldysh.ru/departments/dpt_17/eng/ndeng.htm.

Manson, JoAnn E. et al. 2003. “Estrogen plus progestin and the risk of coronary heart disease.” New England Journal of Medicine 349: 523-534.

Miller, George. 1956. "The magical number seven, plus or minus two: some limits on our capacity for processing information." Psychological Review 63: 81-97.

Miller, Henry and James Bieri. 1963. "An informational analysis of clinical judgment." Journal of Abnormal and Social Psychology 67: 317-325.

Pesaran, M. Hashem and Ron Smith. 1995. “The role of theory in econometrics.” Journal of Econometrics 67: 61-79.

Plous, Scott. 1993. The Psychology of Judgment and Decision Making. New York: McGraw-Hill.

Rimm, Eric B., Meir J. Stampfer, Alberto Ascherio, Edward Giovannucci, Graham A. Colditz, and Walter C. Willett. 1993. “Vitamin E consumption and the risk of coronary heart disease in men.” New England Journal of Medicine 328: 1450-1456.

Smith, Vernon. 2003. “Constructivist and ecological rationality in economics.” American Economic Review 93(3): 465-508.

Sornette, Didier. 2002. Why Stock Markets Crash: Critical Events in Complex Financial Systems. Princeton NJ: Princeton University Press.

Summers, Lawrence H. 1991. “The scientific illusion in empirical macroeconomics.” Scandinavian Journal of Economics 93(2): 129-148.

Vivekananthan, D.P. et al. 2003. “Use of antioxidant vitamins for the prevention of cardiovascular disease: meta-analysis of randomized trials.” Lancet 361:2017-2023.