Goodhart's Law is useless

Other View vs Mainstream View

There is a non-mainstream point of view that "Goodhart's Law is useless". Let's take a look into the pros and cons of this statement:

1. Pros: Goodhart's Law is an oversimplification that fails to account for the intricate complexities of real-world systems and the nuances of human behavior. The mere fact that a measure becomes a target does not inherently mean it loses its value as a measure.
   – Cons (Mainstream View): While Goodhart's Law has its limitations and cannot be applied universally, it nonetheless illuminates a significant phenomenon that manifests frequently when metrics are employed to drive behavior. Numerous real-world examples demonstrate how metrics and incentives can be gamed or lead to unintended and often detrimental consequences.

2. Pros: Organizations and individuals possess the capability to utilize metrics effectively by adopting a balanced scorecard approach, continuously updating and adjusting metrics, and monitoring a diverse array of key performance indicators (KPIs).
   – Cons (Mainstream View): Although there are strategies to mitigate the effects of Goodhart's Law, they require substantial effort and constant vigilance. The forces described by Goodhart remain relevant and problematic across many domains.

3. Pros: Goodhart's Law promotes an excessively fatalistic viewpoint and discourages the use of metrics to drive improvements in systems and performance. Measurement is a fundamental prerequisite for optimization.
   – Cons (Mainstream View): Goodhart's Law does not imply that measurement is futile, but rather emphasizes the need for extreme caution in how metrics are utilized, particularly when tied to incentives. The blind optimization of a metric frequently leads to perverse outcomes.

4. Pros: Goodhart's Law is predicated on the assumption that individuals will invariably act in their own self-interest and endeavor to game the system. In reality, many people possess integrity and will strive to do what is right, even if it does not optimize their personal metrics.
   – Cons (Mainstream View): While not everyone will engage in gaming the system, the incentives created by metric-based targets are incredibly powerful. Even well-intentioned individuals may subconsciously optimize for the metric at the expense of the bigger picture.

5. Pros: Goodhart's Law underestimates the capacity of organizations to develop robust, tamper-proof metrics and monitoring systems that are challenging to game.
   – Cons (Mainstream View): Regardless of how well-designed a metric may be, if it becomes a target, people will devise creative ways to optimize it. It is an ongoing arms race between those designing the metrics and those seeking to game them.

Terms

• Goodhart's Law: An adage stating that "When a measure becomes a target, it ceases to be a good measure." In other words, when a metric is used as a target, people will exploit it, causing it to lose its value as a metric.

• Gaming the system: The act of exploiting the rules, policies, or procedures meant to regulate a system in order to gain an advantage, often to the detriment of the system's intended purpose.

• Perverse incentive: An incentive that generates unintended and undesirable results which are contrary to the intentions of its designers.

Analogy

Goodhart's Law can be likened to a teacher who informs students that they will be evaluated solely based on their test scores. The students may consequently focus on cramming for the test rather than engaging in genuine learning of the material. The test scores, in this case, cease to be a valid measure of the students' knowledge and understanding.

A real-world manifestation of Goodhart's Law can be seen in the Wells Fargo scandal. The bank set aggressive sales targets for its employees, incentivizing them to open as many new accounts as possible. Employees began opening accounts without customer consent, leading to fraudulent activities and severe damage to the bank's reputation. The metric of "new accounts opened" no longer served as a meaningful measure of the bank's health and growth.

History

1. 1975: British economist Charles Goodhart articulates the idea that once a metric becomes a target, it loses its value as a metric, in the context of monetary policy.

2. 1997: Anthropologist Marilyn Strathern restates the law as "When a measure becomes a target, it ceases to be a good measure."

3. 2000s-2010s: Goodhart's Law gains widespread recognition and is frequently invoked in the context of public sector and corporate performance management, as the use of metrics and KPIs becomes increasingly prevalent.

4. 2018: Author David Manheim and researcher Scott Garrabrant propose a generalized version of Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

How to use it

1. When designing an incentive system in a professional setting, it is crucial to consider not only the metrics you aim to optimize but also how individuals may attempt to game those metrics. Employ a combination of quantitative metrics and qualitative evaluations. For example, if measuring customer satisfaction, avoid relying solely on survey scores; take into account customer comments and observe customer interactions to gain a more comprehensive understanding.

2. As an educator, resist the temptation to "teach to the test". Encourage a deeper level of learning and utilize a variety of assessment methods to gauge student progress. For instance, evaluate students' understanding through a combination of tests, essays, projects, and class participation, rather than solely relying on standardized test scores.

3. In the pursuit of personal goals, such as weight loss, avoid fixating solely on a single metric like the number on the scale. Instead, focus on developing healthy, sustainable habits and use multiple measures of progress, such as energy levels, how clothing fits, and overall well-being. Maintain a journal to track diet, exercise, and general wellness, rather than exclusively counting calories or pounds lost.

Facts

• According to a 2018 survey, 55% of U.S. employees believed that performance reviews did not accurately represent their work.

• The "cobra effect" serves as a famous example of Goodhart's Law. During British colonial rule in India, the British government offered a bounty for every dead cobra to reduce the snake population. However, this incentive led to people breeding cobras to collect the bounty, ultimately exacerbating the problem.

• Studies have revealed that when CEO compensation is tied to stock price, CEOs are more likely to engage in financial misreporting and manipulation to artificially inflate the company's stock value.

• In the education sector, an overemphasis on standardized testing has resulted in a narrowing of curricula and a prevalence of "teaching to the test" practices in many schools.

• Some experts argue that Gross Domestic Product (GDP) is no longer an adequate measure of a nation's economic health, as it fails to account for critical factors such as income inequality, environmental sustainability, and quality of life.