S1-404 – What the model missed
Unintended consequences and the price of misapplication
In 1935, the Queensland Bureau of Sugar Experiment Stations imported 101 cane toads from Hawaii and bred them for release into the sugarcane fields of tropical Australia. The decision was based on a model that was simple, plausible, and, within its own terms, coherent. The cane toad had been credited with controlling beetle pests of sugarcane in Puerto Rico, and breeding stock had since been taken to Hawaii, from which Queensland obtained its animals. Queensland had a cane beetle problem. Therefore, Queensland needed cane toads. The model had two components, a causal mechanism connecting each, and a policy recommendation that followed directly. Nothing in the model was obviously wrong. The cane toad does eat beetles. The logic of biological control, using a natural predator to suppress a pest, was established and had worked in other contexts. The decision was reasonable given the information available.
What the model omitted was everything else. The cane toad does not only eat cane beetles. It eats almost anything it can swallow, including native frogs, small reptiles, and the eggs and juveniles of many bird species. It produces a toxin from glands on its back that kills almost any predator that attempts to eat it, including quolls, goannas, and freshwater crocodiles that had no evolutionary history with the toad and therefore no behavioral adaptation to avoid it. It breeds prolifically in any body of standing water. It has no natural predators in Australia that can safely consume it. Nine decades after its introduction, the cane toad population of Australia is estimated at over 200 million. It has spread across most of tropical and subtropical Australia and continues to advance southward. It has caused the collapse of native predator populations across its range. And it was largely ineffective at controlling the cane beetle, which lives at the tops of sugarcane plants while the cane toad feeds on the ground.¹
The cane toad is not a failure of intelligence or of scientific knowledge. It is a failure of model scope: the application of a model whose predictions were accurate within a narrow domain to a context whose relevant features extended far beyond that domain. The model captured the relationship between cane toads and beetles. It had no representation of the relationship between cane toads and the rest of the Queensland ecosystem. The unintended consequences were not random; they were predictable, in retrospect, from the biology of the species and the ecology of the environment into which it was introduced. They were not predicted, in prospect, because the model used to make the decision was not a model of the system. It was a model of one relationship within the system, applied as though the system were nothing more than that relationship.
This article examines five further cases of the same failure (different domains, different scales, different consequences, but the same underlying structure) and draws from them a general account of why complex adaptive systems consistently defeat the models that attempt to manage them.
The mongoose in Hawaii
The cane toad is not unique. It was not even the first case of its kind. In 1883, the small Indian mongoose was introduced to the Hawaiian island of Maui to control rats that were damaging sugarcane crops. The model was identical in structure to the later Queensland decision: rats damage crops, mongooses eat rats, therefore introduce mongooses. The model captured the predator-prey relationship and predicted a reduction in the rat population. It omitted the fact that Hawaiian rats are primarily nocturnal while the mongoose is diurnal: they are active at different times of day and therefore rarely encounter each other. The mongoose, finding rats unavailable, turned to the native ground-nesting birds that had evolved in Hawaii without any mammalian predators and therefore had no behavioral defenses. Several species of native Hawaiian bird were driven to extinction or near-extinction by the mongoose. The rat population was not meaningfully reduced.²
The parallel with the cane toad is exact. In both cases, a two-component model (pest damages crop, predator eats pest) was applied to a context that was a complex ecological system with dozens of interacting species, behavioral specializations, and evolutionary histories that the model had no representation of. In both cases, the intervention produced consequences that were large, lasting, and in the opposite direction from the intended effect for the organism the intervention was supposed to control.
Homo economicus and the retirement savings failure
The biological control failures are dramatic and easy to visualize. The failures that occur when models of human behavior are applied to social and economic systems are less vivid but not less consequential. The most thoroughly documented is the application of the rational agent model, homo economicus, to the design of retirement savings systems.
The rational agent model predicts that individuals, given accurate information about their future needs and the returns available on savings instruments, will make optimal decisions about how much to save and how to invest their savings. The prediction follows directly from the model’s premises: rational agents maximize their expected utility, they have stable time preferences, they can compute the consequences of their choices, and they respond predictably to price signals including interest rates and tax incentives. The policy that follows is also direct: provide accurate information, create tax incentives for retirement savings, and let individuals make their own optimal choices.
The evidence accumulated over decades of behavioral economics research tells a different story. Most people who are given the choice of whether to enroll in a retirement savings plan do not enroll, even when the employer contributes to the plan. Most people who do enroll select the default investment option and never change it, regardless of whether the default is appropriate for their circumstances. Most people dramatically underestimate the amount they need to save for retirement and overestimate the adequacy of social security provisions. The gap between what the rational agent model predicts and what people actually do is not small; it is the difference between adequate retirement provision and old-age poverty for a substantial fraction of any population in which the savings decision is left to individual choice.³
The failure of the rational agent model in this domain is not a failure of the model’s internal logic. The logic is correct given the premises. The failure is a failure of the premises. Real people are not rational agents. They are subject to present bias (the tendency to weight immediate costs more heavily than future benefits), to status quo bias (the tendency to stick with the current arrangement regardless of its optimality), to choice overload (the tendency to make no choice when too many options are available), and to the kind of optimism bias that makes it easy to believe that retirement is far enough away that the decision can be deferred. These features of actual human psychology are not random deviations from the rational model. They are systematic, predictable, and universal across cultures and income levels. A model that treats them as noise rather than signal is a model that will systematically fail to predict the behavior it is supposed to describe.
Pruitt-Igoe and the high-rise housing catastrophe
In 1956, the Pruitt-Igoe housing project opened in St. Louis, Missouri. It consisted of 33 high-rise towers, each 11 stories tall, containing 2,870 units intended to house the low-income residents of the city’s overcrowded slums. The project had been designed by Minoru Yamasaki, who would later design the World Trade Center, according to the modernist principles that dominated architectural thinking of the period: efficient use of space, rational organization of living units, separation of pedestrian and vehicle traffic, communal amenity on the ground floor. The model was the work of serious, thoughtful architects and urban planners who had considered the needs of the residents carefully and designed a system intended to provide them with healthy, dignified, modern accommodation.
By the mid-1960s, Pruitt-Igoe had become one of the most dangerous housing projects in the United States. The communal spaces (the galleries, the skip-stop elevators, the grounds between the towers) had become sites of crime and violence rather than community. The vertical density of the towers had eliminated the informal street-level surveillance that Jane Jacobs would describe in her 1961 book The Death and Life of Great American Cities as the essential mechanism of neighborhood safety. The galleries and elevators were ungoverned spaces, too long and too large for any resident to feel responsible for, too visible from too many angles for any resident to feel safe in. The separation of pedestrian and vehicle traffic, a feature the designers considered a virtue, had eliminated the mixed-use street activity that generates natural social control. By 1972, demolition of the first Pruitt-Igoe towers began. The entire project was razed by 1976, barely 20 years after its opening.⁴
The failure of Pruitt-Igoe was not a failure of technical execution. The buildings were built as designed. The failure was a failure of the social model that the design embodied, a model that treated housing as a technical problem of shelter provision and ignored the social mechanisms through which neighborhoods actually function. The model captured the relationship between living unit density and land use efficiency. It had no representation of the relationship between urban form and the informal social dynamics through which communities produce safety, belonging, and social control. The consequences were predictable from the social dynamics (Jacobs had described them in advance), but the model used to design the project had no category for them.
The opioid crisis as a model failure
The opioid crisis that devastated communities across the United States, and to a lesser degree other countries, from the late 1990s through the 2010s is the most consequential domestic policy failure in American public health in recent decades, and it is a model failure in a specific and identifiable sense.
The model was applied at several levels simultaneously, and it failed at each of them. At the pharmacological level, the model predicted that opioid analgesics, a class of pain medications whose addiction risk had been established for centuries, would not produce addiction when prescribed for chronic non-cancer pain, because the pain itself would consume the drug’s reward potential. This prediction was based on a misapplication of findings from acute pain settings to chronic pain settings: patients given opioids in hospital for acute post-surgical pain did indeed show low rates of addiction, but this finding did not transfer to chronic pain patients in community settings, who faced the full addictive properties of the drugs without the protective context of acute suffering. The pharmacological model was applied outside its domain of valid calibration.
At the regulatory level, the model predicted that physician prescribing would be constrained by professional norms and clinical judgment to appropriate medical uses, without requiring the regulatory burden that would impede access for patients with genuine pain needs. The model failed to account for the pressure that pharmaceutical company marketing (specifically the aggressive marketing of OxyContin by Purdue Pharma, with explicit claims about the drug’s non-addictive properties) would place on prescribing norms, and for the inability of regulatory structures designed for acute care to monitor the cumulative effects of prescribing patterns at the population level. The regulatory model was applied without representation of the commercial dynamics that would distort the clinical dynamics it assumed.
At the social level, the model predicted that addiction, if it occurred, would be managed within existing treatment infrastructure. It failed to account for the way in which the simultaneous prescription of opioids to large populations in communities with high unemployment, economic distress, and social fragmentation would create the conditions for addiction to spread through social networks: through the sharing of prescribed medications between family members, through the diversion of pharmaceuticals to street markets, through the transition from pharmaceutical opioids to cheaper heroin when prescriptions became harder to obtain. The social model was applied without representation of the community-level dynamics that would transform individual clinical decisions into a population-level catastrophe.⁵
Structural adjustment and the development model
The structural adjustment programs imposed on developing countries by the International Monetary Fund and World Bank during the 1980s and 1990s represent the most large-scale application of an economic model to development policy in the late twentieth century, and one of the most thoroughly documented failures of model misapplication in any domain.
The model was derived from the Washington Consensus: a set of policy recommendations centered on fiscal discipline, trade liberalization, privatization of state enterprises, deregulation of markets, and the removal of barriers to foreign investment. The model had genuine intellectual grounding: each of its components was supported by economic theory and by evidence from some contexts. Fiscal discipline reduces inflation. Trade liberalization increases efficiency. Privatization removes the inefficiencies of state management. The predictions followed from well-established economic mechanisms.
What the model omitted was the institutional context without which the mechanisms do not operate as predicted. Trade liberalization increases efficiency when the country has the institutional capacity to compete in liberalized markets, when it has functioning courts, enforceable contracts, access to credit for firms, and workers who can move to more productive sectors. When it lacks these institutions, trade liberalization destroys domestic industry without creating the alternative employment that the model assumes. Privatization reduces inefficiency when the privatized firms operate in competitive markets with effective regulatory oversight. When they operate in uncompetitive markets with weak regulation, privatization replaces state inefficiency with private extraction. Fiscal discipline stabilizes economies when it is applied at the right moment in the economic cycle. When applied during a recession, it deepens the recession by reducing demand and constraining the countercyclical fiscal response that recovers it.
The structural adjustment programs were applied across dozens of countries with different institutional histories, different levels of development, different economic structures, and different political contexts, applied through a template that had no representation of these differences, because the model treated the mechanisms as universal and the institutional context as irrelevant. The consequences included severe recessions, dramatic increases in inequality, destruction of domestic manufacturing capacity, and, in several cases, political instability and humanitarian crises that required emergency intervention. The model was not wrong about the mechanisms. It was wrong about the conditions under which the mechanisms operate as predicted.⁶
The common structure
Six case studies, six domains, six different scales of consequence, but a single underlying structure of failure. In every case, a model that captured a real mechanism within a specific domain was applied to a context whose relevant features extended far beyond that domain. In every case, the model had no representation of the features of the broader context that would determine whether the mechanism would operate as predicted or produce different and in some cases opposite effects. And in every case, the unintended consequences were not random; they were predictable, in retrospect, from the features of the context that the model had omitted.
The common structure has three components. The first is component-model reasoning: the application of a model built on a specific, isolated mechanism to a system that contains many other mechanisms that interact with the first. The cane toad model was a model of a predator-prey relationship. The Queensland ecosystem was a system of hundreds of species with thousands of interacting relationships, of which the predator-prey relationship between toads and beetles was one. The rational agent model was a model of individual decision-making under idealized conditions. The retirement savings system was a social and institutional context with behavioral, cognitive, and social dynamics that the model had no representation of. In each case, the model was a component model applied to a system.
The second component is interaction blindness: the failure to model the interactions between the targeted mechanism and the other mechanisms in the system. The cane toad model did not ask: how will the introduction of a new predator affect the other predators in the system? The rational agent model did not ask: how will individuals respond to the choice architecture of the savings system rather than to its expected utility? The structural adjustment model did not ask: how will trade liberalization interact with the absence of functioning credit markets? The interactions were not predicted because the model had no representation of them.
The third component is adaptive response: the tendency of the elements of complex systems, particularly human social systems, to respond to the model’s interventions in ways that change the conditions the model assumed. This is the mechanism that the sociologist Robert Merton identified as the self-defeating prophecy and that the economist Charles Goodhart captured in the law named after him: when a measure becomes a target, it ceases to be a good measure. When a model is used to design an intervention, the intervention changes the system in ways that the model predicts, but it also changes the system in ways that the elements of the system generate in response to the intervention, responses that the model, having been built before the intervention, has no representation of.⁷
Three principles for navigating complexity
The history of unintended consequences does not recommend paralysis. Complex systems must be managed, and management requires models. What it recommends is a specific set of practices for managing with models in conditions where the model’s scope is known to be narrower than the system’s actual dynamics.
The first is provisionality: treating interventions as experiments rather than implementations. An experiment is designed to generate information about whether the model’s predictions are correct. An implementation is designed to achieve an outcome that the model has established will follow from the intervention. The difference is not merely semantic. An experiment includes a monitoring system capable of detecting when the model’s predictions are wrong, a feedback mechanism capable of feeding the information back to the decision-maker, and a reversibility structure capable of adjusting or abandoning the intervention when the monitoring reveals unexpected outcomes. Biological control programs conducted today, unlike the 1935 cane toad release, are preceded by extensive risk assessment and are conducted on a small scale with intensive monitoring before any decision to expand is made.
The second is monitoring with genuine detection power: the capacity to detect failures that the model did not predict, not merely to measure whether the predicted outcomes are occurring. A monitoring system designed around a model can only detect the outcomes the model predicts. It is blind, by construction, to the outcomes the model did not predict, which are precisely the unintended consequences that matter. Genuine monitoring requires observation of the system beyond the model’s predictions: systematic attention to outcomes that were not expected, to changes in indicators that the model treats as stable, and to signals from parts of the system that the model treats as irrelevant.
The third is reversibility: the preference, wherever possible, for interventions whose consequences can be undone over interventions whose consequences are permanent. The cane toad cannot be removed from Australia. The communities destroyed by opioid addiction cannot have the addiction reversed by removing the drugs. The manufacturing capacity destroyed by premature trade liberalization cannot be quickly reconstructed. These irreversible consequences are not unique to complex systems (some simple interventions are also irreversible), but complex systems have a specific tendency to produce irreversible consequences, because the interactions between components can produce changes in system structure that are path-dependent: the system moves to a new equilibrium that is stable and that cannot be reversed by undoing the original intervention. The preference for reversibility is not a preference against intervention. It is a preference for maintaining the option to learn and adjust, the option that irreversible interventions foreclose.
Further reading
Jane Jacobs’s The Death and Life of Great American Cities (1961) is the foundational text for understanding what the urban planning models of the mid-twentieth century missed, the informal social dynamics of neighborhood life that the modernist model treated as irrelevant and that turned out to be the primary mechanism of urban safety and community. It is the most important book on urban planning ever written and remains directly relevant to how cities are designed and managed.
James C. Scott’s Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed (1998) provides the most thorough available analysis of the common structure of large-scale planned interventions that have produced catastrophic unintended consequences, arguing that the failure in each case is the application of a simplified, legible model to a complex, illegible reality, and that the simplification required to make the model manageable is the simplification that destroys the features of the system that make it work.
Nassim Nicholas Taleb’s Antifragile: Things That Gain from Disorder (2012) develops the complementary positive argument: that systems designed to be robust to the unintended consequences of model misapplication share specific structural features (redundancy, optionality, small experiments rather than large commitments, preference for convex over concave exposure to uncertainty) and that these features can be deliberately cultivated.
Barry Commoner’s The Closing Circle: Nature, Man, and Technology (1971) articulates the foundational ecological principle that underlies the biological control cases: in a complex ecosystem, everything is connected to everything else, and interventions that target one connection without modeling the others will produce effects through the untargeted connections that overwhelm the intended effect on the targeted one.
For the specific case of opioids, Patrick Radden Keefe’s Empire of Pain: The Secret History of the Sackler Dynasty (2021) provides the most thorough available account of the commercial and regulatory dynamics that the clinical model of opioid prescribing had no representation of, the specific mechanisms by which pharmaceutical marketing distorted clinical judgment at scale.
Notes
¹ The cane toad (Rhinella marina) introduction to Queensland is documented in Lever, C. (2001). The Cane Toad: The History and Ecology of a Successful Colonist. Westbury Academic and Scientific Publishing. The original introduction in 1935 was made by the Bureau of Sugar Experiment Stations following visits by Queensland entomologists to Puerto Rico, where cane toads had been introduced for beetle control. The effectiveness of the Puerto Rico introduction, which the Queensland model was based on, was itself later questioned. The current distribution and population estimate of cane toads in Australia is maintained by the Australian government’s Invasive Species Council. The predator toxicity problem arises because the cane toad’s parotoid glands produce bufadienolides, cardiac glycosides that produce rapid cardiac arrest in most vertebrates that consume them. Species that evolved with toads (such as those in the Americas) have behavioral adaptations to avoid them. Australian species did not.
² The mongoose introduction to Hawaii is documented in Tomich, P. Q. (1986). Mammals in Hawaii. Bishop Museum Press. The diurnal-nocturnal mismatch between the mongoose and Hawaiian rats was known to some scientists at the time of introduction but was not considered in the decision. The mongoose subsequently spread to all the main Hawaiian islands except Kauai (where it was apparently absent from the initial introduction) and Lanai. It is now considered one of the most significant threats to Hawaiian ground-nesting seabirds and is responsible for the near-extinction of several species including the Hawaiian petrel and the Newell’s shearwater.
³ The behavioral economics literature on retirement savings decisions is reviewed in Thaler, R. H., and Benartzi, S. (2004). Save More Tomorrow: Using behavioral economics to increase employee saving. Journal of Political Economy, 112(S1), S164-S187. The most important single finding is that automatic enrollment in savings plans, with an opt-out rather than opt-in structure, dramatically increases participation rates without changing any of the economic parameters of the plan. This finding is the practical expression of the nudge argument: that behavior can be changed more reliably by changing the choice architecture than by changing the economic incentives, because the behavioral features that the rational agent model ignores (status quo bias, present bias, choice overload) are more powerful determinants of behavior than the economic incentives the model captures.
⁴ Jacobs, J. (1961). The Death and Life of Great American Cities. Random House. Jacobs’s analysis of what she called “eyes on the street,” the informal surveillance of public space by residents going about their daily business, was a direct critique of the modernist urban planning model that produced Pruitt-Igoe. Her argument was that the mixed-use street with varied building types, short blocks, and ground-floor activity generated a continuous social presence that deterred crime and produced community, while the superblock high-rise tower isolated residents from the street and from each other in ways that destroyed the informal mechanisms of neighborhood safety. The demolition of Pruitt-Igoe in 1972-76 is sometimes cited as the moment of modernist architecture’s death, though the architectural historian Charles Jencks’s precise dating of the movement’s death to 3:32 PM on July 15, 1972, when the first Pruitt-Igoe towers were dynamited, is slightly theatrical.
⁵ The opioid crisis in the United States is analyzed from multiple perspectives in Dasgupta, N., Beletsky, L., and Ciccarone, D. (2018). Opioid crisis: No easy fix to its social and economic determinants. American Journal of Public Health, 108(2), 182-186. The specific role of Purdue Pharma’s marketing of OxyContin, including the company’s promotion of the false claim that the drug’s extended-release formulation made it non-addictive, is documented in Van Zee, A. (2009). The promotion and marketing of OxyContin: Commercial triumph, public health tragedy. American Journal of Public Health, 99(2), 221-227.
⁶ The structural adjustment programs of the IMF and World Bank in the 1980s and 1990s are assessed in Stiglitz, J. E. (2002). Globalization and Its Discontents. W. W. Norton. Stiglitz, a former Chief Economist of the World Bank and Nobel laureate in economics, provides an insider account of the institutional dynamics that produced the template-application approach (including the time pressure of crisis conditions, the career incentives of IMF staff, and the political dynamics of the creditor-debtor relationship) that makes the model-application failure understandable as an institutional product rather than merely an intellectual error. The empirical evidence on the outcomes of structural adjustment programs is reviewed in Przeworski, A., and Vreeland, J. R. (2000). The effect of IMF programs on economic growth. Journal of Development Economics, 62(2), 385-421.
⁷ Goodhart’s Law, “when a measure becomes a target, it ceases to be a good measure,” was first articulated by the economist Charles Goodhart in a 1975 paper on monetary policy: Goodhart, C. A. E. (1975). Problems of monetary management: The UK experience. Papers in Monetary Economics (Reserve Bank of Australia), 1. The law captures the specific form of adaptive response that is most consequential for social policy: the tendency of agents in a system to optimize for the measured indicator rather than for the underlying outcome the indicator is supposed to track, thereby destroying the correlation between the indicator and the outcome. The classic example is the proliferation of teaching to the test when educational accountability systems use standardized test scores as performance indicators, producing improvements in measured outcomes that do not translate to improvements in actual educational achievement.
