S1-402 – The unmodelable
Chaos, randomness, and the limits of prediction
On the morning of October 15, 1987, Michael Fish, a senior meteorologist at the BBC, broadcast the following reassurance to the British public: “Earlier on today, apparently, a woman rang the BBC and said she heard there was a hurricane on the way. Well, if you’re watching, don’t worry, there isn’t.” That night, the Great Storm of 1987 struck southern England and northern France with winds exceeding 100 miles per hour, the most violent storm to hit the region in nearly three centuries. Eighteen people died. Fifteen million trees were felled. Insurance losses ran to billions of pounds. The meteorological models available to the forecasters had not predicted it. The storm had developed faster than the models could track, from conditions that the models had assessed as benign. Michael Fish was not incompetent. The models were not primitive. The system being modeled, the atmosphere over the North Atlantic, was simply behaving in ways that were, in the deepest possible sense, beyond the reach of the available predictions.
The 1987 storm is often cited as an example of forecasting failure, an embarrassment to be remedied by better models, more data, and faster computers. This is partly true: weather forecasting has improved substantially since 1987, and modern forecast models would have given several days’ warning of the Great Storm. But the improvement is bounded. There is a ceiling on the predictability of atmospheric systems, not a ceiling imposed by the limitations of current computers or current models, but a ceiling imposed by the mathematics of the systems themselves. Some things are unpredictable not because we lack the tools to predict them but because unpredictability is a structural feature of the systems that generate them.
This article examines two distinct and fundamentally different sources of that unpredictability, chaos and randomness, and argues that understanding the difference between them is one of the most practically important things that an educated person can know about the limits of models.
Two kinds of unpredictability
The distinction between chaos and randomness is frequently blurred in popular discourse, where both words are used interchangeably to mean “disordered” or “hard to predict.” The distinction is, however, precise and important, and the two phenomena have entirely different implications for what models can and cannot do.
Randomness, in the strict sense, is the absence of any causal determination of an outcome. A genuinely random event has no prior cause that, if known, would allow the event to be predicted. The outcome is not merely unknown; it is unknowable, in principle and not merely in practice, because there is no fact of the matter about what it will be until it occurs. Randomness in this strict sense is found, as far as the current understanding of physics extends, at the quantum level: the specific moment at which a radioactive atom decays, the specific direction in which a photon is scattered, the specific outcome when a quantum system is measured. These events are not random because we lack the information to predict them. They are random because quantum mechanics, the best-confirmed theory in the history of physics, tells us that no additional information could make them predictable. The universe, at the quantum level, is genuinely, irreducibly stochastic.¹
Chaos is entirely different. A chaotic system is deterministic: its future state is fully determined by its current state, and there is in principle a correct prediction of everything it will do. What makes chaotic systems unpredictable in practice is an exquisite sensitivity to initial conditions: tiny differences in the starting state of the system, differences far too small to measure with any available instrument, produce enormous differences in the system’s future behavior, and these differences grow exponentially with time. The atmosphere is a chaotic system. If the atmosphere were in exactly the same state today as it was on some previous occasion, it would evolve in exactly the same way, and the same weather would follow. But the atmosphere is never in exactly the same state: there are always differences, arbitrarily small, and those differences amplify as the system evolves until the predictions based on slightly different initial states are entirely different. The unpredictability of weather is not because weather is random. It is because weather is chaotic.²
The distinction matters enormously for what we should conclude from the fact of unpredictability. Quantum randomness is irreducible: no improvement in measurement, no increase in computational power, no refinement of theory can make the specific outcome of a quantum event predictable, because there is no fact of the matter about the outcome that a better theory or instrument could capture. Chaotic unpredictability is, in principle, reducible (if we could measure initial conditions with arbitrary precision and compute with arbitrary speed, chaotic systems would be perfectly predictable), but the reduction is in practice impossible, because the required precision grows exponentially with the prediction horizon. Beyond a certain time horizon, the required precision exceeds anything available or achievable with any conceivable instrument.
The butterfly effect and its consequences
The chaotic sensitivity of atmospheric systems was first characterized mathematically by the meteorologist Edward Lorenz in the early 1960s. Lorenz was running a computer simulation of atmospheric convection, a simplified model of the kind of fluid dynamics that underlies weather, when he made an observation that changed the history of both meteorology and mathematics. He had stopped and restarted a simulation from the middle, intending to reproduce the second half of a run he had already made. To save time, he entered the starting values by hand from a printout, typing 0.506 for a value that the computer had stored internally as 0.506127. The difference was less than one part in a thousand. The simulation diverged dramatically from the original within a short time, producing an entirely different pattern of behavior.
Lorenz recognized that this was not a computing error. It was a fundamental property of the system he was modeling: small differences in initial conditions produced large differences in outcomes, and the growth was exponential. The phenomenon acquired the name that has made it famous, the butterfly effect, from the title of a 1972 paper in which Lorenz asked: does the flap of a butterfly’s wings in Brazil set off a tornado in Texas? The answer is not that it does. The answer is that the question cannot be answered: in a chaotic system, the contribution of any specific small perturbation to any specific large outcome is in principle untraceable, because the perturbation interacts with every other perturbation in ways that cannot be computed forward without exact knowledge of every other perturbation.³
The practical limit on weather prediction that the butterfly effect implies is well established: beyond approximately two weeks, the uncertainties in any weather forecast, however well-constructed the model and however good the measurements, grow to the point where the forecast has essentially no skill. This is not a contingent limitation of current technology. It is a mathematical consequence of the structure of the atmosphere. Improvements in model resolution and initial condition measurement have pushed this limit outward (the useful skill of forecasts has increased from a few days in 1987 to approximately 10 days currently), but the limit exists and is not movable beyond approximately two weeks for global weather prediction, regardless of the improvement in models or computing power.
The same mathematics applies, with different time constants, to every other chaotic system. Population dynamics in ecology: populations of predators and prey can evolve chaotically, with small differences in initial conditions producing wildly different long-term trajectories. Economic systems: small perturbations in consumer sentiment, investment decisions, or policy choices can interact in ways that produce large aggregate differences, making long-term economic forecasting subject to the same fundamental limits as weather forecasting. And traffic flow: the familiar phenomenon of phantom traffic jams, congestion that appears without any obvious external cause, that propagates backward through traffic against the direction of travel, is a consequence of the chaotic dynamics of a dense stream of vehicles, each making small independent decisions that interact to produce collective instability.
Quantum indeterminacy: randomness at the foundation
The randomness that quantum mechanics describes is not merely a practical limitation on measurement. It is, according to the most thoroughly tested theory in the history of physics, a fundamental feature of reality. The electron does not have a definite position and a definite momentum simultaneously, not because we cannot measure them both at the same time, but because the quantum state of the electron does not include definite values for both simultaneously. The Heisenberg uncertainty principle is not a statement about measurement disturbance. It is a statement about the structure of quantum states: the more precisely a quantum state specifies one observable, the less precisely it can specify the conjugate observable.
The philosophical implications of this have been debated since the 1920s without reaching consensus. The Copenhagen interpretation, associated primarily with Niels Bohr, treats the quantum state as a complete description of reality and accepts that measurement outcomes are genuinely random, irreducibly probabilistic, with no hidden variables that could in principle make them predictable. The Everett many-worlds interpretation avoids the randomness by postulating that all possible outcomes occur, in different branches of a universal wave function, so that from inside any branch the outcome appears random but from outside there is a deterministic evolution of the whole. The pilot wave theory of de Broglie and Bohm preserves determinism by postulating hidden variables that guide particle trajectories, but the hidden variables are in principle inaccessible. Bell’s theorem, proved in 1964 and confirmed experimentally many times since, establishes that no theory of the type that preserves both locality (causes cannot propagate faster than light) and determinism can reproduce the predictions of quantum mechanics.⁴ Whatever the correct interpretation of quantum mechanics, the predictions, the probability distributions for measurement outcomes, are the same, and those predictions have been confirmed to extraordinary precision. The randomness, whatever its ultimate interpretation, is real and irreducible at the level of observable predictions.
The practical consequences of quantum randomness at the macroscopic level are usually negligible: quantum uncertainty is so small relative to the scales of everyday objects that it plays no direct role in the prediction of their behavior. But there are specific circumstances where quantum randomness propagates to macroscopic scales and becomes practically significant. Radioactive decay, which is quantum-mechanical in origin, produces genuine randomness in the timing of nuclear events that is directly relevant to nuclear engineering and medical physics. Quantum effects in biological systems, including the possibility that certain aspects of photosynthesis, bird navigation, and olfaction exploit quantum coherence, are an active area of research. And the most significant consequence: in a chaotic system, quantum uncertainty provides a floor to the uncertainty in initial conditions that cannot in principle be reduced below the Heisenberg limit, which means that chaotic systems are doubly unpredictable: uncertain in their initial conditions due to quantum indeterminacy, and exponentially sensitive to those uncertainties due to chaos.
The black swan: the unpredictable from outside the model
There is a third source of predictive failure that is distinct from both chaos and randomness and that Nassim Nicholas Taleb has named and analyzed in his account of black swan events:⁵ the event that falls outside the model not because the model is wrong within its domain but because the event comes from outside the domain entirely.
A financial risk model built on the historical volatility of liquid markets does not predict a liquidity crisis (an event in which buyers and sellers simultaneously withdraw from a market, making it impossible to determine a price at all), because liquidity crises are not part of the statistical distribution from which the model was estimated. The model was estimated on data from periods when markets functioned normally. The liquidity crisis is precisely the condition under which markets stop functioning, and it is therefore the condition most systematically excluded from the data on which models are calibrated.
The black swan is not random in the quantum sense and not chaotic in the Lorenz sense. It is something different: an event that the model cannot represent, not because of sensitivity to initial conditions but because the model has no category for it. The black swan is the unknown unknown of article S1-204 applied to quantitative modeling: not a measurable uncertainty within the model’s framework, but an event that falls entirely outside the framework. The model’s prediction for a black swan event is not wrong. It is absent: the event is not in the model’s sample space, and the model therefore cannot assign it a probability at all.
This third source of predictive failure is in some ways the most consequential, because it is the most invisible. Chaotic unpredictability is at least visible as unpredictability: the forecast horizon is known, the uncertainty grows in ways that can be estimated, and users of the forecast know that the forecast becomes unreliable beyond a certain point. Quantum randomness is visible as an irreducible probability distribution: the theory tells us exactly how probable each outcome is, and the randomness is therefore quantified and communicable. The black swan is invisible from within the model: from inside the model, there is no signal that an event outside the model’s domain is approaching, because the model has no representation of the domain boundary.
What the limits of prediction imply for The Conscious Look
The existence of genuine, irreducible limits on prediction, from quantum randomness, from chaos, and from black swans, has specific and important implications for the practice this series recommends.
The first implication is that uncertainty is not always reducible by more information. The weather beyond two weeks is unpredictable not because we lack data but because the system is chaotic. The specific outcome of a quantum event is unpredictable not because we lack a theory but because the theory tells us there is no fact of the matter to know. The black swan is unpredictable not because we have failed to gather the relevant data but because the relevant data comes from a domain the model does not represent. In each case, the appropriate response to the uncertainty is not the demand for more information, which will not reduce the uncertainty, but the calibration of decisions to the irreducible uncertainty that remains. Decisions that cannot tolerate the uncertainty of weather beyond two weeks should not be made on the basis of weather forecasts extending beyond two weeks. Decisions that cannot tolerate black swan events should be structured with the reversibility and redundancy that makes survival of such events possible.
The second implication is that the confidence expressed in a model’s predictions should reflect the source of the uncertainty. A weather forecast for tomorrow is highly reliable: the atmosphere is chaotic, but the prediction horizon is short enough that the exponential growth of uncertainty has not yet made the forecast skill negligible. A 10-day forecast is considerably less reliable. A 20-day forecast has essentially no skill. The source of the uncertainty, chaos, implies a specific structure to the confidence that should be placed in forecasts of different horizons. Similarly, a financial model that estimates the probability of a loss within the historical range of market conditions can be highly reliable within that range, while being entirely uninformative about losses that fall outside the historical range, the black swan territory. The model’s confidence should be expressed as conditional on remaining within its calibration domain, and the probability of leaving that domain, the probability of a black swan, is precisely what the model cannot estimate.
The third implication is that some questions should not be answered with models. The meteorologist who provides a 20-day weather forecast is not giving a useful prediction. They are providing a false sense of certainty that may be more harmful than acknowledged ignorance. The financial analyst who provides a probability estimate for an event that has never occurred in the historical data is providing a number that has no meaningful relationship to the actual probability. The appropriate response to questions that lie beyond the predictive reach of available models is not to extend the model beyond its valid domain. It is to acknowledge the limit and to structure the decision accordingly, with the humility about irreducible uncertainty that the mathematics of chaos, randomness, and black swans demands.
The world is not fully predictable. It is not fully predictable for reasons that are now well understood: the quantum structure of matter, the chaotic dynamics of complex systems, and the existence of events that fall outside every model’s domain. Understanding these reasons is not an invitation to despair about knowledge. It is the condition of holding knowledge honestly: of knowing which questions models can answer and which they cannot, and of making decisions with the calibrated confidence that the actual state of knowledge warrants.
Further reading
James Gleick’s Chaos: Making a New Science (1987) is the most readable account available of the discovery and development of chaos theory, following Lorenz’s initial observation through the mathematical development of the theory by a sequence of researchers in different disciplines, and conveying both the intellectual excitement of the discovery and its practical implications. It does not require a mathematical background and has not dated.
Nassim Nicholas Taleb’s The Black Swan: The Impact of the Highly Improbable (2007) provides the most sustained available analysis of the third source of predictive failure, the event that falls outside the model’s domain, and its consequences for risk management, financial modeling, and decision-making under genuine uncertainty. Taleb is polemical and sometimes imprecise, but the core argument is important and is made with unusual force.
David Deutsch’s The Fabric of Reality (1997) provides the most philosophically serious available treatment of the Everett many-worlds interpretation of quantum mechanics, the interpretation that preserves determinism by postulating a branching universe, and its implications for the nature of knowledge and explanation. It is demanding but rewarding.
Roger Penrose’s The Emperor’s New Mind (1989) argues, controversially, that quantum randomness plays a specific and important role in the generation of consciousness, and that this role implies that certain aspects of human thought are not computable. The argument is contested but the treatment of quantum mechanics and its philosophical implications is authoritative.
For the specific practical implications of chaos theory for weather prediction, Tim Palmer’s work on ensemble forecasting, representing uncertainty in weather prediction through a collection of forecasts from slightly different initial conditions, is the most accessible technical development of how operational weather forecasting has incorporated the lessons of chaos theory. A good introduction is Palmer, T. N. (2000). Predicting uncertainty in forecasts of weather and climate. Reports on Progress in Physics, 63(2), 71-116.
Notes
¹ The claim that quantum randomness is irreducible, that no theory of hidden variables could restore determinism at the quantum level, is more precisely stated as follows: Bell’s theorem (1964) proves that no local hidden variable theory can reproduce the statistical predictions of quantum mechanics. Experiments by Aspect and colleagues in 1982, and by many subsequent groups in increasingly loophole-free designs, have confirmed the quantum mechanical predictions and excluded local hidden variable theories. Non-local hidden variable theories (such as the pilot wave theory of de Broglie and Bohm) can reproduce the quantum mechanical predictions deterministically, but they require non-local influences, correlations between distant events that cannot be used for signaling but that nonetheless violate the spirit of locality. The practical conclusion, that quantum randomness cannot be reduced by more information about the system, holds for all interpretations of quantum mechanics, because all interpretations agree on the statistical predictions.
² The mathematical definition of chaos requires three properties: determinism (the future state is a function of the current state), sensitivity to initial conditions (nearby trajectories diverge exponentially), and topological mixing (the system is not decomposable into non-interacting subsystems). The sensitivity to initial conditions is measured by the Lyapunov exponent, the average rate of exponential divergence of nearby trajectories. A positive Lyapunov exponent is the signature of chaotic behavior, and its magnitude determines the prediction horizon: the time after which the uncertainty in a prediction has grown to the size of the attractor (the typical scale of the system’s behavior). For the atmosphere, the Lyapunov exponent implies a prediction horizon of approximately two weeks. Lorenz, E. N. (1963). Deterministic nonperiodic flow. Journal of the Atmospheric Sciences, 20(2), 130-141.
³ The butterfly effect metaphor originated in the title of a talk that Lorenz delivered to the American Association for the Advancement of Science in Washington in 1972: “Predictability: Does the Flap of a Butterfly’s Wings in Brazil Set Off a Tornado in Texas?” The title was suggested by the session chairman, Philip Merilees, and Lorenz adopted it as a vivid expression of the sensitivity to initial conditions that his mathematical work had established. The original paper is available at: Lorenz, E. N. (1972). Predictability: Does the flap of a butterfly’s wings in Brazil set off a tornado in Texas? Address to the American Association for the Advancement of Science. The metaphor is sometimes misread as claiming that specific butterfly wing-flaps cause specific tornadoes, a causal claim that the mathematics does not support. The correct reading is that differences as small as a butterfly’s wing-flap can, in principle, change the state of the atmosphere sufficiently to produce a different long-term outcome, but which specific outcome any specific perturbation produces is itself beyond prediction.
⁴ Bell, J. S. (1964). On the Einstein Podolsky Rosen paradox. Physics, 1(3), 195-200. Bell’s theorem is one of the most important results in twentieth-century physics. The Einstein-Podolsky-Rosen paradox, formulated in 1935, argued that if quantum mechanics is complete (if there are no hidden variables), then measurements on entangled particles must exhibit instantaneous correlations across arbitrary distances, a conclusion that Einstein found unacceptable (“spooky action at a distance”). Bell showed that the predictions of any local hidden variable theory differ from the predictions of quantum mechanics in a way that is experimentally testable, the Bell inequalities, and that the experimental tests have consistently confirmed the quantum mechanical predictions and violated the Bell inequalities. The most loophole-free tests are described in Hensen, B., et al. (2015). Loophole-free Bell inequality violation using electron spins separated by 1.3 kilometres. Nature, 526(7575), 682-686.
⁵ Taleb, N. N. (2007). The Black Swan: The Impact of the Highly Improbable. Random House. Taleb’s core argument is that human beings and human institutions systematically underestimate the probability of events that fall outside their historical experience, because they model the world using statistical distributions estimated from past data, and past data by definition excludes events that have not yet occurred. The financial risk management community’s reliance on Value at Risk (VaR) models estimated from historical market data is his primary target, but the argument applies equally to any model that is calibrated on data from a restricted domain and then used to assess risks that include events outside that domain. The mathematical framework underlying Taleb’s argument, extreme value theory and the distinction between thin-tailed and fat-tailed distributions, is developed more technically in Mandelbrot, B., and Hudson, R. L. (2004). The (Mis)behavior of Markets: A Fractal View of Financial Turbulence. Basic Books.