Building Less Flawed Metrics

Manheim, David (2018): Building Less Flawed Metrics.

Abstract

Metrics are useful for measuring systems and motivating behaviors. Unfortunately, naive application of metrics to a system can distort the system in ways that undermine the original goal. The problem was noted independently by Campbell and Goodhart, and in some forms it is not only common, but unavoidable due to the nature of metrics. There are two distinct but interrelated problems that must be overcome in building better metrics; first, specifying metrics more closely related to the true goals, and second, preventing the recipients from gaming the difference between the reward system and the true goal. This paper describes several approaches to designing metrics, beginning with design considerations and processes, then discussing specific strategies including secrecy, randomization, diversification, and post-hoc specification. Finally, it will discuss important desiderata and the trade-offs involved in each approach.

• Item Type: MPRA Paper
• Language: English
• Keywords: Metrics, Goodhart's Law, Campbell's Law, Cobra Effect, Management, Control Theory
• Subjects: D - Microeconomics > D8 - Information, Knowledge, and Uncertainty; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D80 - General; I - Health, Education, and Welfare > I2 - Education and Research Institutions > I26 - Returns to Education; I - Health, Education, and Welfare > I2 - Education and Research Institutions > I28 - Government Policy; J - Labor and Demographic Economics > J4 - Particular Labor Markets > J48 - Public Policy; Z - Other Special Topics > Z1 - Cultural Economics ; Economic Sociology ; Economic Anthropology > Z18 - Public Policy

References

[Cam79] Donald T Campbell. Assessing the impact of planned social change. Evaluation and program planning, 2(1):67–90, 1979.

[Cap18] Bryan Caplan. The case against education. Why the education system is a waste of time and money. Princeton University Press, 2018.

[CHF+16] Martin Cames, Ralph O Harthan, J¨urg F¨ussler, M Lazarus, C Lee, P Erickson, and Randall Spalding-Fecher. How additional is the clean development mechanism. Analysis of application of current tools and proposed alternatives. Oeko-Institut EV CLlMA. B, 3, 2016.

[CP14] Helene L Caudill and Constance D Porter. An Historical Perspective of Reward Systems: Lessons Learned from the Scientific Management Era. International Journal of Human Resource Studies; Vol 4, No 4 (2014)DO - 10.5296/ijhrs.v4i4.6605, dec 2014.

[Der15] William Deresiewicz. Excellent sheep: The miseducation of the American elite and the way to a meaningful life. Free Press, 2015.

[FBBAK18] Laura Fraade-Blanar, Marjory S Blumenthal, James M Anderson, and Nidhi Kalra. Measuring Automated Vehicle Safety. 2018.

[Goo75] Charles A E Goodhart. Problems of monetary management: the UK experience. In Papers in Monetary Economics. Reserve Bank of Australia, 1975.

[Her68] Frederick Herzberg. One more time: How do you motivate employees, 1968.

[Hes18] Frederick Hess. Straight Up Conversation: Scholar Jay Greene on the Importance of Field Trips. Education Week, sep 2018.

[Hub14] Douglas W. Hubbard. How to Measure Anything: Finding the Value of Intangibles in Business. LIVRO, 2014.

[Kle07] Gary Klein. Performing a project premortem. Harvard Business Review, 85(9):18–19, 2007.

[LK18] Stan Liebowitz and Matthew L. Kelly. Everything You Know About State Education Rankings Is Wrong: Minds and dollars are a terrible thing to waste. Reason, nov 2018.

[Man16] David Manheim. Overpowered Metrics Eat Underspecified Goals. Ribbonfarm.com, 2016.

[Man18] David Manheim. Shorrock’s Law of Limits. Medium.com, May 2018. 14

[MEP89] Deborah J Mitchell, J Edward Russo, and Nancy Pennington. Back to the future: Temporal perspective in the explanation of events. Journal of Behavioral Decision Making, 2(1):25–38, 1989.

[MG18] David Manheim and Scott Garrabrant. Categorizing Variants of Goodhart’s Law. pages 1–10, 2018.

[Mul18] Jerry Z Muller. The tyranny of metrics. Princeton University Press, 2018.

[PP16] Konstantinos Poulis and Efthimios Poulis. Problematizing fit and survival: transforming the law of requisite variety through complexity misalignment. Academy of Management Review, 41(3):503–527, 2016.

[RM01] Jonathan Rosenhead and John Mingers. Rational analysis for a problematic world revisited. Number 2nd. John Wiley and Sons, 2001.

[Rod17] Jeffery Rodamar. There ought to be a law! Campbell v. Goodhart. 2017.

[RR17] Imran Rasul and Daniel Rogger. Management of bureaucrats and public service delivery: Evidence from the nigerian civil service. The Economic Journal, 128(608):413–446, 2017.

[RRW17] I Rasul, D Rogger, and M Williams. Management and bureaucratic effectiveness: A scientific replication. 2017.

[RRW18] Imran Rasul, Daniel Rogger, and Martin J. Williams. Autonomy, incentives, and the effectiveness of bureaucrats. VoxDev, 2018.

[Sim47] Herbert A Simon. Administrative behavior; a study of decision-making processes in administrative organization. Macmillan, 1947.

[Str97] Marilyn Strathern. ’Improving ratings’: audit in the British University system. European Review, 1997.

[TC12] Dana H Taplin and H`elene Clark. Theory of change basics: A primer on theory of change. 2012.