Munich Personal RePEc Archive

Building Less Flawed Metrics: Dodging Goodhart and Campbell's Laws

Manheim, David (2018): Building Less Flawed Metrics: Dodging Goodhart and Campbell's Laws. Forthcoming in:

This is the latest version of this item.

[img] PDF
MPRA_paper_90649.pdf

Download (190kB)
[img]
Preview
PDF
MPRA_paper_98288.pdf

Download (242kB) | Preview

Abstract

Metrics are useful for measuring systems and motivating behaviors. Unfortunately, naive application of metrics to a system can distort the system in ways that undermine the original goal. The problem was noted independently by first Campbell, then Goodhart, and in some forms it is not only common, but unavoidable due to the nature of metrics. There are two distinct but interrelated problems that must be overcome in building better metrics; first, specifying metrics more closely related to the true goals, and second, preventing the recipients from gaming the difference between the reward system and the true goal. This paper describes several approaches to designing metrics, beginning with design considerations and processes, then discussing specific strategies including secrecy, randomization, diversification, and post-hoc specification. The discussion will then address important desiderata and the trade-offs involved in each approach, and examples of how they differ, and how the issues can be addressed. Finally, the paper outlines a process for metric design for practitioners who need to design metrics, and as a basis for further elaboration in specific domains.

Available Versions of this Item

UB_LMU-Logo
MPRA is a RePEc service hosted by
the Munich University Library in Germany.