S1-306 – The widening gap
How we learned to use the world without understanding it
At some point in the last decade, the device most of us are reading these words on became something we cannot explain. Not merely something we could not build (that has been true of most tools for most of human history) but something whose operating principles are beyond the reach of the vocabulary we use every day to describe what it does. We know that touching a glass surface produces text on a screen. We do not know why. We know that the device communicates wirelessly with other devices across the planet. We do not know how. We know that the device contains something called a processor, something called memory, something called an operating system, and something called an app. We do not know what any of these things actually are, in the sense of being able to describe the physical and logical processes that make them function. We use the words. The words are not understanding.
This is not a personal failing. It is a structural condition of modern life, and it is accelerating. The gap between what human beings can use and what they can understand has been widening for two centuries, driven by the progressive division of knowledge into specializations that are individually comprehensible but collectively beyond the grasp of any single mind. Each generation inherits a world that is more capable, more connected, and less intelligible than the one before it. It is a world in which the tools available for navigation have multiplied while the capacity to understand those tools has remained bounded by the cognitive limits that article S1-202 described: the small working memory, the chunking mechanism, the knowledge illusion that makes familiarity feel like understanding. We have built a civilization whose infrastructure is, in the technical sense, beyond us, and we live in it with extraordinary ease, right up to the moment when something goes wrong.
This article examines the four stages through which the gap between using and understanding has widened, what the gap costs in specific domains, and what can be done (not to close it, which is neither possible nor desirable, but to hold it honestly).
The four stages of the widening gap
The gap between using a tool and understanding it is not new. But its scale, depth, and consequences have changed in ways that are worth tracing through the specific stages at which the gap widened most dramatically.
In the first stage (the stage that characterized most of human history before industrialization) the relationship between user and tool was essentially transparent. The hunter who used a spear, the farmer who used a plough, the weaver who used a loom, could in principle build the tool they used, repair it when it broke, understand the physical principles by which it worked, and assess the consequences of its use on the environment in which they used it. The tool was comprehensible to the person who used it across all four dimensions: construction, operation, mechanism, and impact. This is not to romanticize pre-industrial life (it was difficult in ways that the tool transparency did not compensate for) but to identify the baseline: a condition in which the gap between using and understanding was small enough that the user’s model of the tool was approximately accurate in all four dimensions.
In the second stage (the stage of early industrialization, roughly the eighteenth and nineteenth centuries) the first dimension separated from the others. The worker who operated a steam engine could not build one. Construction had become specialist. But operation, mechanism, and impact remained accessible: the worker who operated the engine could understand how it worked in functional terms, could recognize when it was malfunctioning, and could observe the consequences of its operation in the environment. The gap had opened in one dimension. Three remained transparent.
In the third stage (the stage of mature industrialization, running from roughly the late nineteenth century through the twentieth) the second dimension separated. The worker who operated increasingly complex machinery could operate it without understanding how it worked. The user of a refrigerator, a car, or a television operated the device through an interface designed to make operation possible without mechanical understanding. The mechanism had become specialist. Construction and mechanism were now both beyond the ordinary user. Operation and impact remained accessible: one could still use the device and observe what it did to the world around it. The gap had widened to two dimensions.
In the fourth stage (the stage we are currently in, which began approximately with the personal computer and has accelerated with networked devices, platforms, and algorithmic systems) the fourth dimension has separated as well. The user of a smartphone, a social media platform, or an AI-driven recommendation system cannot build it, does not understand its mechanism, operates it through an interface designed to conceal its operation, and increasingly cannot directly observe its impact on the environment: on their own psychology, their social relationships, their information environment, and the broader social and political systems in which they participate. All four dimensions of the original transparency have closed. The device is usable and opaque across every dimension simultaneously.¹
Alfred North Whitehead observed that civilization advances by extending the number of important operations which we can perform without thinking about them. This is true and important. The operations that the fourth stage has made performable without thinking include communication across continents, access to the accumulated knowledge of human civilization, and the coordination of social and economic activity at a scale that no previous generation could have imagined. The price of these operations, in Whitehead’s terms, is precisely that they do not require thinking (do not require understanding) and that the absence of understanding is therefore invisible to the people performing the operations. The gap is not felt. It is lived.
What the gap costs in specific domains
The costs of the widening gap are not distributed evenly across domains. They are concentrated in three specific areas where the consequences of not understanding the tools one uses are most consequential.
The first is the political domain. Democratic self-governance requires that citizens be able to evaluate the claims of those who seek to govern them, to assess the likely consequences of proposed policies, and to hold their representatives accountable for the actual consequences of the policies enacted. This evaluation requires some understanding of the systems being governed: the economic system, the public health system, the energy system, the information system. As these systems have become more complex, more interconnected, and more opaque, the gap between what citizens are asked to evaluate and what they can understand has widened correspondingly. The voter who is asked to form a view on monetary policy, epidemiological modeling, energy transition pathways, or algorithmic content regulation is being asked to evaluate the outputs of systems whose mechanisms they cannot directly access. They must rely on intermediaries (experts, journalists, politicians, commentators) whose own understanding of the systems varies enormously and whose incentives to communicate that understanding accurately vary equally.²
The second domain is personal decision-making. The tools that now mediate the most consequential personal decisions (about health, about finance, about relationships, about information) are increasingly algorithmic and therefore increasingly opaque. The search engine that determines what information is available to us when we are making a medical decision does not show us how it ranks results or whose interests its ranking reflects. The credit scoring algorithm that determines our access to financial products does not explain which features of our history it weighted and why. The platform that determines which of our friends’ posts we see does not tell us on what basis it makes that selection or how the selection affects our political and social beliefs over time. We use the outputs of these systems to make decisions. We do not understand the systems. And the decisions we make on the basis of their outputs are therefore decisions made on the basis of processes whose biases and errors we cannot assess.³
The third domain is the one the series’ outline identified as this article’s distinctive contribution: the engineering and environmental domain. The fourth stage of the widening gap has separated not only construction and mechanism from ordinary understanding but impact: the ability to directly observe the consequences of the tools one uses on the environment in which one lives. The consequences of the digital economy (on attention, on mental health, on social cohesion, on political polarization, on energy consumption) are real and large, but they are not directly observable in the way that the consequences of an early industrial factory were observable. The factory smoked visibly. The smartphone does not.
This fourth-dimension opacity is the specific feature of the current stage that makes it qualitatively different from previous stages. In all previous stages, the impact dimension remained transparent even as the other three closed. The person who could not build a car, did not understand its engine, and operated it through an interface that concealed its mechanism could still observe that it produced exhaust, that it damaged pedestrians when driven into them, and that its proliferation changed the character of urban space. The impact was visible and therefore available for democratic deliberation. The impact of the current generation of digital tools (their effects on individual psychology, social dynamics, political systems, and information environments) is not visible in the same way, because it operates through mechanisms that are themselves opaque, produces effects that are gradual, distributed, and statistically rather than individually visible, and occurs in a domain (the interior life of the person using the tool) that is inherently private.⁴
The knowledge illusion at civilizational scale
The widening gap between using and understanding is the civilizational version of the knowledge illusion described in article S1-202. The knowledge illusion at the individual level is the confusion of familiarity with understanding: the sense that because one can recognize a concept, one understands it. The knowledge illusion at the civilizational level is the confusion of capability with comprehension: the sense that because civilization can do something, civilization understands what it is doing.
The pencil example from article S1-202 is relevant here: no single person on earth knows how to make a pencil, yet pencils are made in their billions. The distributed knowledge that makes pencil production possible does not reside in any individual mind. It resides in the network of specialists, institutions, supply chains, and accumulated practices that together constitute the production system. The same is true, at vastly greater scale and complexity, of every system that characterizes modern civilization. The distributed knowledge that makes the internet function does not reside in any individual mind or any single institution. It is distributed across a network of specialists whose individual domains of understanding are narrow relative to the whole, and whose collective production constitutes something that no individual understands.
This distributed knowledge is genuinely powerful (it is what has produced the extraordinary material capabilities of the current era) and it is also genuinely fragile in ways that Perrow’s normal accidents analysis (discussed in article S1-204) predicts: complex, tightly coupled systems fail in ways that no individual component anticipated, because the failure modes arise from interactions between components rather than from the failure of any individual component. The fragility of the systems that constitute modern civilization is not visible to their users, because the fourth stage of the gap has made impact opacity a structural feature of the tools themselves. The systems work, until they don’t. And when they don’t, the opacity that made them usable is the same opacity that makes the failure difficult to diagnose and correct.
The specialist and the citizen: a structural problem
The widening gap creates a structural problem for democratic societies that has no fully satisfactory solution but that can be managed with greater or lesser wisdom depending on how clearly it is understood. The problem is this: the systems that democratic governance is asked to regulate are increasingly beyond the understanding of the citizens who must govern them, which means that democratic governance must increasingly rely on the expertise of specialists who understand the systems, and who therefore have interests, incentives, and blind spots that the democratic process cannot easily evaluate.
The expert whose knowledge is necessary for governance has power that is not subject to the ordinary accountability mechanisms of democracy, because the citizen cannot assess the quality of the expert’s judgment without the understanding that makes the expert’s knowledge expert. This is not a new problem (it has existed in various forms throughout the history of democratic governance) but it has become more acute as the gap has widened and as the systems being governed have become more complex, more interconnected, and more consequential.
The response to this problem that the series’ framework recommends is not a return to first-stage transparency: that is neither possible nor desirable, and the attempt to achieve it would require dismantling the complexity that has produced the capabilities we most value. It is the combination of two things: the cultivation, at the individual level, of the epistemic literacy that article S1-213 described (the ability to assess the quality of expert claims, identify the open questions and internal disagreements within a discipline, and apply the Feynman test to distinguish genuine understanding from fluent vocabulary) and the development, at the institutional level, of the structures that make expert knowledge more accountable to democratic oversight without requiring that every citizen become a specialist.⁵
The second of these (institutional accountability for expert knowledge) is the subject of later upcoming of this series, which examine the specific mechanisms by which expertise enters democratic governance and the specific ways those mechanisms can fail. What this article contributes to that discussion is the prior observation: the gap is structural, it is widening, and the appropriate response to it begins with acknowledging it honestly rather than with the pretense that the tools one uses are more transparent than they are.
The Conscious Look, applied to the tools we use
The practice this series recommends, applied to the widening gap, is not a demand for universal technical understanding: that is impossible and would be, in the fourth stage, counterproductive even if it were possible. It is the practice of knowing, with as much precision as possible, in which of the four dimensions one’s understanding of any given tool is adequate and in which it is not.
This means knowing, for any tool that mediates a significant decision or shapes a significant aspect of one’s life: can I build it? The answer for almost all contemporary tools is no, and that is acceptable. Do I understand its mechanism, the physical or logical process by which it produces its outputs from its inputs? The answer for many contemporary tools is also no, but this is less acceptable, because understanding the mechanism is often what allows one to assess the reliability of the outputs and to recognize when the tool is failing or being used inappropriately. Do I understand its interface, the logic by which the tool presents itself to me and the choices that interface embeds? The answer should more often be yes, and when it is not, the deficit is worth addressing. And do I understand its impact, the effects of my use of the tool on myself, on other people, and on the broader systems in which I participate? This is the hardest question, because impact opacity is the characteristic feature of the fourth stage, but it is also the most important question, because impact is what the tool ultimately does to the world.
The widening gap is not going to close. The complexity of the systems that constitute modern civilization is not going to decrease. The appropriate response is not nostalgia for the first stage (the hunter who could build their own spear) but the honest acknowledgment of where one stands relative to the tools one uses, and the cultivation of enough understanding of mechanism and impact to use those tools responsibly rather than merely conveniently.
Further reading
Steven Sloman and Philip Fernbach’s The Knowledge Illusion: Why We Never Think Alone (2017) provides the cognitive foundation for the civilizational argument of this article: the mechanism by which distributed knowledge produces individual illusions of understanding, and the consequences of those illusions for personal and political decision-making.
Matthew Crawford’s Shop Class as Soulcraft: An Inquiry into the Value of Work (2009) makes the case for the specific value of manual competence (of work that maintains a direct relationship between action and consequence, between understanding and use) as a counterweight to the first-stage transparency that the fourth stage has eliminated. It is the most readable available treatment of what is lost when the gap widens, stated as a personal and philosophical argument rather than a sociological analysis.
Shoshana Zuboff’s The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power (2019) provides the most thorough available treatment of the impact opacity argument: the specific ways in which the behavioral consequences of using digital tools are invisible to users while being highly visible to the platforms that harvest and monetize them. It is long and demanding but essential for understanding the fourth dimension of the current stage.
Frank Pasquale’s The Black Box Society: The Secret Algorithms That Control Money and Information (2015) examines the specific mechanisms by which algorithmic systems mediate consequential decisions (in finance, in employment, in information access) without accountability to the people whose lives they shape, and proposes specific institutional responses to the opacity problem.
Charles Perrow’s Normal Accidents: Living with High-Risk Technologies (1984), discussed in article S1-204, provides the theoretical foundation for understanding why the complexity and tight coupling of fourth-stage systems makes them structurally prone to failures that their users cannot anticipate, and why the impact opacity of those failures is not an accident but a structural feature of the systems themselves.
Notes
¹ The four-stage framework developed here is an analytical construct rather than a historical periodization, and the stages overlap considerably in time and across different technologies and societies. Many technologies currently in use span multiple stages simultaneously: agricultural tools remain largely in the first stage, mechanical tools largely in the second and third, and digital tools largely in the fourth. The framework is not a claim about when specific technologies appeared but about the structure of the relationship between users and tools (which dimensions of transparency are preserved and which have closed) and how that structure has changed across the development of modern technology.
² The political consequences of the widening gap are examined in the context of specific policy domains in article S1-808 of this series, which addresses the specific challenge of climate change as a multi-domain problem that democratic governance is asked to address without the understanding that such governance ideally requires. The argument made here (that the gap is structural and widening) is the background against which the more specific political arguments of the society and politics pillar should be read.
³ The opacity of algorithmic decision-making is not only a practical problem for the individuals whose decisions are mediated by algorithms. It is a legal and constitutional problem: if a decision about credit, employment, or criminal sentencing is made by an algorithm whose reasoning cannot be inspected, the procedural protections that democratic legal systems provide for consequential decisions (the right to know the basis on which a decision was made, the right to contest the decision, the right to present evidence to a human decision-maker) are effectively nullified. The legal and regulatory responses to this problem are at an early stage and are developed in the context of article S1-612 of this series, which examines the specific capabilities and limitations of artificial intelligence as a decision-making tool.
⁴ The impact opacity of digital tools on mental health is the most extensively studied current instance of the fourth-dimension gap. The research literature on the effects of social media use on psychological wellbeing, particularly among adolescents, is large and contested: contested not because the effects do not exist but because their magnitude, directionality, and causal mechanisms are genuinely difficult to establish when the exposure is pervasive, gradual, and varies enormously across individuals and platforms. The most thorough review of the evidence is Haidt, J., and Rausch, Z. K. (2023). Social media and mental health: A collaborative review. Available at jonathanhaidt.com. The contested nature of the evidence is itself an instance of the point being made: the impact is real but not directly observable, which makes its assessment depend on exactly the kind of statistical and methodological sophistication that the widening gap makes increasingly unavailable to non-specialists.
⁵ The specific epistemic literacy practices described in article S1-213 (the open problems probe, the expert disagreement test, the Feynman explanation test, the adjacent field test, the historical depth test) are the individual-level responses to the structural problem of the widening gap. They cannot close the gap, but they can make the person using them more accurate about where they stand relative to it: more honest about which dimensions of the tools they use they understand and which they do not, and more alert to the specific ways in which impact opacity operates in the domains that matter most to them.