How to Choose the Level of Significance: A Pedagogical Note

Kim, Jae (2015): How to Choose the Level of Significance: A Pedagogical Note.

Abstract

The level of significance should be chosen with careful consideration of the key factors such as the sample size, power of the test, and expected losses from Type I and II errors. While the conventional levels may still serve as practical benchmarks, they should not be adopted mindlessly and mechanically for every application.

• Item Type: MPRA Paper
• Language: English
• Keywords: Expected Loss, Statistical Significance, Sample Size, Power of the test
• Subjects: A - General Economics and Teaching > A2 - Economic Education and Teaching of Economics; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General

References

DeGroot, M. 1975. Probability and Statistics, 2nd ed. Reading, MA: Addison-Wesley.

DeGroot, M. H. and Schervish, M. J., 2012, Probability and Statistics, 4th edition, Addison-Wesley, Boston

Engsted, T. 2009, Statistical vs. economic significance in economics and econometrics: Further comments on McCloskey and Ziliak, Journal of Economic Methodology, 16, 4, 393-408.

Fomby, T. B. Guilkey, D. K., 1978, On Choosing the Optimal Level of Significance for the Durbin-Watson test and the Bayesian alternative, Journal of Econometrics, 8, 203-213.

Gigerenzer, G. 2004, Mindless statistics: Comment on “Size Matters”, Journal of Socio-Economics, 33, 587-606.

Johnson, V. E. 2013, Revised standards for statistical evidence, Proceedings of the National Academy of Sciences, www.pnas.org/cgi/doi/10.1073/pnas.1313476110

Keuzenkamp, H.A. and Magnus, J. 1995, On tests and significance in econometrics, Journal of Econometrics, 67, 1, 103–128.

Kim, J. H. and Ji, P. 2015, Significance Testing in Empirical Finance: A Critical Review and Assessment, Journal of Empirical Finance 34, 1-14.

Kim, J. H. and Choi, I. 2016, Unit Roots in Economic and Financial Time Series: A Re-Evaluation at the Optimal Level of Significance: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2700659

Kish, L. 1959, Some statistical problems in research design, American Sociological Review, 24, 328-338.

Labovitz, S. 1968, Criteria for selecting a significance level: a note on the sacredness of 0.05, The American Sociologist, 3, 200-222.

Leamer, E. 1978, Specification Searches: Ad Hoc Inference with Nonexperimental Data, Wiley, New York.

Lehmann E.L. and Romano, J.S. 2005, Testing Statistical Hypothesis, 3rd edition, Springer, New York.

Manderscheid, L.V., 1965, Significance Levels-0.05, 0.01, or ?, Journal of Farm Economics, 47 (5), 1381-1385.

Moore, D.S. and McCabe, G.P. 1993, Introduction to the Practice of Statistics, 2nd edition, W.H. Freeman and Company, New York.

Morrison, D. E. and Henkel, R. E. 1970, The Significance Test Controversy: A Reader, edited by D. E. Morrison and R. E. Henkel. Aldine Transactions, New Brunswick, NJ.

Skipper, J. K. JR., Guenther, A. L. and Nass, G. 1967, The sacredness of .05: a note on concerning the use of statistical levels of significance in social science, The American Sociologist, 2, 16-18.

Soyer, E. and Hogarth, R. M. 2012, The illusion of predictability: How regression statistics mislead experts, International Journal of Forecasting, 28, 695-711.

Wasserstein R. L., Lazar, N. A. 2016, The ASA's statement on p-values: context, process, and purpose, The American Statistician, DOI: 10.1080/00031305.2016.1154108

Winer, B. J. 1962, Statistical Principles in Experimental Design, New York, McGraw-Hill.

Ziliak, S. T. and McCloskey, D.N. 2008, The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives, Ann Arbor, The University of Michigan Press.