Omitted-variable bias and other matters in the defense of the category adjustment model: A reply to Crawford (2019)

Duffy, Sean and Smith, John (2019): Omitted-variable bias and other matters in the defense of the category adjustment model: A reply to Crawford (2019).

Abstract

The datasets from Duffy, Huttenlocher, Hedges, and Crawford (2010) [Psychonomic Bulletin & Review, 17(2), 224-230] were reanalyzed by Duffy and Smith (2018) [Psychonomic Bulletin & Review, 25(5), 1740-1750]. Duffy and Smith (2018) conclude that the datasets are not consistent with the category adjustment model (CAM). Crawford (2019) [Psychonomic Bulletin & Review, 26(2), 693-698] offered a reply to Duffy and Smith (2018) that is based on three main points. Crawford proposes regressions that are, in part, based on a “deviation” analysis. Crawford offers a different simulation of data and claims that the techniques employed by Duffy and Smith (2018) are not sufficiently sensitive to detect a specific relationship that is claimed to be consistent with CAM. Crawford also appeals to a figure showing that the responses appear to be biased toward the overall running mean, and presumably not toward recently viewed lines. We show that Crawford’s analysis suffers from an omitted-variable bias. Once this bias is corrected, the evidence in support of CAM disappears. When we produce a simulated dataset that is consistent with the specification suggested by Crawford, the techniques of Duffy and Smith (2018) correctly detect the true relationship. Despite the assertion otherwise, the simulated dataset that was analyzed by Crawford is not publicly available. Since the analysis of Crawford (2019) is incorrect, it remains our view that the datasets from Duffy, Huttenlocher, Hedges, and Crawford (2010) do not appear to be consistent with CAM or any Bayesian model of judgment.

• Item Type: MPRA Paper
• Language: English
• Keywords: judgment, omitted-variable bias, category adjustment model, central tendency bias, recency effects, Bayesian judgments
• Subjects: C - Mathematical and Quantitative Methods > C9 - Design of Experiments > C91 - Laboratory, Individual Behavior

References

Blackwell, D., & Dubins, L. (1962). Merging of opinions with increasing information. Annals of Mathematical Statistics, 33, 882-886.

Crawford, L. E. (2019). Reply to Duffy and Smith’s (2018) reexamination. Psychonomic Bulletin & Review, 26(2), 693-698.

Duffy, S., Huttenlocher, J., Hedges, L. V., & Crawford, L. E. (2010). Category effects on stimulus estimation: Shifting and skewed frequency distributions. Psychonomic Bulletin & Review, 17(2), 224-230.

Duffy, S., & Smith, J. (2018). Category effects on stimulus estimation: Shifting and skewed frequency distributions—A reexamination. Psychonomic Bulletin & Review, 25(5), 1740-1750.

Duffy, S., & Smith, J. (2019). On the Category Adjustment Model: Another look at Huttenlocher, Hedges, and Vevea (2000). Working paper, Rutgers University-Camden.

Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70(3), 193-242.

Hill, R.C., Griffiths, W.E., & Judge, G.G. (2001). Undergraduate Econometrics. Wiley and Sons: New York.

Hollingworth, H. L. (1910). The central tendency of judgment. The Journal of Philosophy, Psychology and Scientific Methods, 7(17), 461-469.

Huttenlocher, J., Hedges, L. V., & Vevea, J. L. (2000). Why do categories affect stimulus judgment? Journal of Experimental Psychology: General, 129, 220-241.

Savage, L.J. (1954). The Foundations of Statistics. Wiley, New York. Reprinted in 1972 by Dover, New York.