Verification of Citations: Fawlty Towers of Knowledge?

Wright, Malcolm and Armstrong, J. Scott (2007): Verification of Citations: Fawlty Towers of Knowledge? Forthcoming in: Interfaces

Abstract

The prevalence of faulty citations impedes the growth of scientific knowledge. Faulty citations include omissions of relevant papers, incorrect references, and quotation errors that misreport findings. We discuss key studies in these areas. We then examine citations to Estimating nonresponse bias in mail surveys, one of the most frequently cited papers from the Journal of Marketing Research, as an exploratory study to illustrate these issues. This paper is especially useful in testing for quotation errors because it provides specific operational recommendations on adjusting for nonresponse bias; therefore, it allows us to determine whether the citing papers properly used the findings. By any number of measures, those doing survey research fail to cite this paper and, presumably, make inadequate adjustments for nonresponse bias. Furthermore, even when the paper was cited, 49 of the 50 studies that we examined reported its findings improperly. The inappropriate use of statistical-significance testing led researchers to conclude that nonresponse bias was not present in 76 percent of the studies in our sample. Only one of the studies in the sample made any adjustment for it. Judging from the original paper, we estimate that the study researchers should have predicted nonresponse bias and adjusted for 148 variables. In this case, the faulty citations seem to have arisen either because the authors did not read the original paper or because they did not fully understand its implications. To address the problem of omissions, we recommend that journals include a section on their websites to list all relevant papers that have been overlooked and show how the omitted paper relates to the published paper. In general, authors should routinely verify the accuracy of their sources by reading the cited papers. For substantive findings, they should attempt to contact the authors for confirmation or clarification of the results and methods. This would also provide them with the opportunity to enquire about other relevant references. Journal editors should require that authors sign statements that they have read the cited papers and, when appropriate, have attempted to verify the citations.

• Item Type: MPRA Paper
• Institution: Ehrenberg-Bass Institute, University of South Australia
• Language: English
• Keywords: citation errors; evidence-based research; nonresponse bias; quotation errors; surveys
• Subjects: Y - Miscellaneous Categories > Y8 - Related Disciplines; C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology ; Computer Programs > C81 - Methodology for Collecting, Estimating, and Organizing Microeconomic Data ; Data Access; B - History of Economic Thought, Methodology, and Heterodox Approaches > B4 - Economic Methodology

References

Armstrong, J. S. 2007. Significance tests harm progress in forecasting. Inter. J. Forecasting 23 (2007) 321-327. Armstrong, J. S. 1996. Management folklore and management science: On portfolio planning, escalation bias, and such. Interfaces 26(4) 28–42. Armstrong, J. S., T. S. Overton. 1977. Estimating nonresponse bias in mail surveys. J.Marketing Res. 14(3) 396–402. Aronsky, D., J. Ransom, K. Robinson. 2005. Accuracy of references in five biomedical informatics journals. J. Amer. Medical Informatics Association 12(2) 225 –228. Collier, J. E., C. C. Bienstock. 2007. An analysis of how nonresponse error is assessed in academic marketing research. Marketing Theory 7(2) 163–183. DeLacey, G, C. Record, J. Wade. 1985. How accurate are quotations and references in medical journals? British Medical J. 291 884–886. L. Schulmeister. Quotation and reference accuracy of three nursing journals. J. Nursing Scholarship 30(2) 143 –147. Dillman, D. A. 2000, Mail and Internet Surveys: The Tailored Design Method, 2nd ed. New York, John Wiley & Sons. Doms, C. A. 1989. A survey of reference accuracy in five national dental journals. J. Dental Res. 68(3) 442–444. Eichorn P., A. Yankauer. 1987. Do authors check their references? A survey of accuracy of references in three public health journals. Amer. J. Public Health. 77(8) 1011–12. Evans J. T., H. I. Nadjari, S. A. Burchell. 1990. Quotational and reference accuracy in surgical journals: A continuing peer review problem. J. Amer. Medical Association. 263(10) 1353–54. Fenton, J. E., H. Brazier, A. De Souza, J. P. Hughes, D. P. McShane. 2000. The accuracy of citation and quotation in otolaryngology/head and neck surgery journals. Clinical Otolaryngology 25 40 –44.

Franke, R. H. 1980. Worker productivity at Hawthorne. Amer. Sociol. Rev. 45(6) 1006–1027. Franke, R. H. 1996. Advancing knowledge, a comment from Richard H. Franke. Interfaces 26(4) 52–57. Franke, R. H., J. D. Kaul. 1978. The Hawthorne experiments: First statistical interpretation. Amer. Sociol. Rev. 43(5) 623–643. Gendall, P. 2000. Responding to the problem of nonresponse. Australasian J. Marketing Res. 8(1) 3–18. Gosling, C. M., M. Cameron, P. F. Gibbons. 2004. Referencing and quotation accuracy in four manual therapy journals. Manual Therapy 9 36 –40. Lok, C. K.W., M. T. V. Chan, I. M. Martinson. 2001. Risk factors for citation errors in peer-reviewed nursing journals. J. Adv. Nursing 34(2) 223–229. Lukic, I. K., A. Lukic, V. Gluncic, V. Katavic, V. Vucenik, A. Marusic. 2004. Citation and quotation accuracy in three anatomy journals. Clinical Anatomy 17 534–539. MacRoberts, M. H., B. R. MacRoberts. 1986. Quantitative measures of communication in science: A study of the formal level. Social Stud. Sci. 16 151 –172. MacRoberts, M. H., B. R. MacRoberts . 1988. Problems of citation analysis: A critical review. J. Amer. Soc. Inform. Sci. 40(5) 342 –249. Parsons, H. M. 1974. What happened at Hawthorne?" Sci.183 922–932. Roach, V. J., T. K. Lau, D. Warwick, N. Kee. 1997. The quality of citations in major international obstetrics and gynecology journals. Amer. J. Obstetrics and Gynecology 177(4) 973 –975. Roethlisberger, F. J., W. J. Dickson. 1939. Management and the Worker. Harvard University Press: Cambridge, MA. Schulmeister, L. 1998. Quotation and reference accuracy of three nursing journals. Image: J. Nursing Scholarship 30(2) 143 –147. Wager, E., P. Middleton. 2003. Technical editing of research reports in biomedical journals. The Cochrane Database of Methodology Reviews. 1 Art. No. MR000002, DOI: 10.1002/14651858 MR000002.