211 – Merit beliefs and crony beliefs

Why the strength of a conviction is not evidence of its truth

Think about the last time you genuinely changed your mind about something important. Not refined your position, not updated a minor detail, but actually reversed a significant belief: concluded that what you had been confident about was wrong and adopted the opposing view. If you are like most people, the occasions are few and the memory of each one is distinct. Now think about how it happened. Almost certainly not through a calm weighing of evidence that produced a gradual shift in probability estimates. Almost certainly through something more disruptive: an experience that could not be explained by the existing belief, an encounter with a person whose intelligence you respected and whose position you had not expected, a moment of private honesty in which the gap between what you said you believed and what you actually found yourself acting on became too large to ignore.

The rarity of genuine belief revision and the specific conditions under which it occurs are two of the most important and least discussed facts about human cognition. We spend an enormous amount of time forming, defending, and expressing beliefs. We spend very little time revising them. And when revision does occur, it is typically not produced by the kind of evidence-weighing that the rational model of belief formation assumes. It is produced by the kinds of events that disrupt the social and psychological functions that the belief was serving, functions that had nothing to do with tracking truth.

This article is about those functions. It draws primarily on a framework developed by the writer and software engineer Kevin Simler in a 2016 essay titled “Crony Beliefs,” which offers the most precise available account of why the beliefs we hold most strongly are often the beliefs we have examined least rigorously, and why that relationship is not accidental.

The company analogy

Simler proposes an analogy that is worth developing at length because its explanatory power justifies the space it requires. Imagine a company that has two kinds of employees: merit employees and crony employees. Merit employees were hired because they are genuinely good at the job: they produce value, they are accountable to measurable performance standards, and they would be replaced if they consistently failed to deliver. Crony employees were hired because of their relationships with powerful people inside the company: they are there because of who they know, not what they can do, and they persist in their positions not because of their performance but because of the social costs of removing them.¹

Most well-run companies have mostly merit employees and a few cronies. Poorly run companies, or companies operating in environments where competitive pressure is absent, accumulate cronies until the proportion of merit employees is insufficient to sustain the organization’s actual functions. The company continues to exist, continues to generate revenue, continues to look from the outside like a functioning organization, but its internal culture has been corrupted by a class of employees whose continuation is secured by political rather than performative means.

Simler applies this analogy to beliefs. Merit beliefs are the beliefs that are held because they track truth: beliefs whose justification has been examined, whose predictions have been tested against experience, and whose continuation in the belief system is secured by their accuracy. Crony beliefs are the beliefs that are held because of their social and psychological function, beliefs that are there not because the evidence supports them but because they serve the interests of the person holding them: their social identity, their tribal affiliations, their emotional needs, their professional commitments, their self-concept. Like crony employees, crony beliefs persist not because of their performance (their accuracy, their predictive success, their explanatory power) but because of the social and psychological costs of removing them.

The distinction is important because it explains something that the standard account of belief (as a representation of how things are, updated by evidence) cannot explain: why the beliefs we hold most strongly are often the beliefs we have examined least carefully, why the beliefs most resistant to revision are typically the most consequential, and why intelligent, thoughtful people maintain beliefs that are inconsistent with the evidence available to them not because they have not seen the evidence but because the evidence is not what is doing the work of maintaining the belief.

What crony beliefs look like from the inside

The crucial and uncomfortable property of crony beliefs is that they are, from the inside, indistinguishable from merit beliefs. The person holding a crony belief does not experience it as a belief held for social or psychological reasons. They experience it as a belief held because it is true: as a genuine representation of how things are, supported by reasons that feel compelling and evidence that feels decisive. The cognitive experience of crony belief is identical to the cognitive experience of merit belief. The difference is in the causal history and the functional role, not in the phenomenology.

This is why the standard approach to challenging crony beliefs (presenting contrary evidence, pointing out logical inconsistencies, demonstrating the belief’s failure to predict) is so reliably ineffective. The evidence and the argument are being evaluated by a cognitive system that is not, in this domain, primarily in the business of tracking truth. It is in the business of defending a position whose social and psychological function makes it resistant to revision. The evaluation is not neutral. The deck is stacked. And the stacking is invisible to the person doing the evaluation, because the belief feels like a merit belief (feels like it is there because it is true) even when it is there for entirely different reasons.

The phenomenological indistinguishability of crony beliefs from merit beliefs means that introspection is not a reliable guide to the question of which category a given belief falls into. The feeling of conviction is not evidence. The sense that one has good reasons is not evidence. The experience of having thought carefully about the question is not evidence. What is needed is an external diagnostic: a way of assessing the belief’s functional role that does not rely on introspective access to the processes that produced it.

The diagnostic: four hallmarks of crony beliefs

Simler proposes a set of diagnostic features that distinguish crony beliefs from merit beliefs with reasonable reliability. They are not infallible, but they identify a cluster of properties that, when present together, strongly suggest that a belief is being maintained for non-epistemic reasons.

The first hallmark is abstractness. Crony beliefs tend to be abstract in ways that make them difficult to test against specific predictions. The belief that markets are fundamentally fair, or that human nature is fundamentally cooperative, or that a particular political party serves the interests of ordinary people: these beliefs can accommodate almost any specific observation, because they are pitched at a level of generality that keeps them insulated from specific disconfirmation. Merit beliefs, by contrast, tend to generate specific predictions that could in principle be wrong. The abstractness of a belief is not conclusive evidence of cronyism (some abstract beliefs are well-supported and important) but a belief that cannot be cashed out in specific predictions deserves scrutiny.

The second hallmark is emotional protection. Crony beliefs are surrounded by a characteristic emotional response to challenge: not the engaged curiosity of someone whose belief is being tested, but the discomfort, anxiety, or anger of someone whose social or psychological interests are under threat. The challenge does not feel like an interesting argument to be examined. It feels like an attack. This emotional response is itself informative, because it suggests that the belief is connected to something more important than its truth value (to identity, belonging, or self-concept) and that the defense being mounted is not primarily an epistemic defense.

The third hallmark is resistance to betting. Simler notes that crony beliefs are typically beliefs their holders would be unwilling to bet on.² This is a specific and revealing diagnostic because betting requires putting something at stake that the believer actually values (money, reputation, a concrete commitment) and because it forces the question of how confident one actually is, as distinct from how confident one presents oneself as being. The person who confidently asserts that a political party will win an election, that an investment will succeed, that a policy will produce the desired outcome, but who is unwilling to bet meaningfully on that assertion, has revealed something important: their expressed confidence is not backed by actual confidence, which suggests that the belief is performing a function other than representing their genuine probability estimate.

The fourth hallmark is the gap between public expression and private behavior. Crony beliefs are typically held for their social function (for what they signal to the community of people who share them) and this means they are often expressed more confidently in public than they are acted on in private. The environmentalist who advocates strongly for carbon restrictions but who has not made the significant lifestyle changes that the urgency of their stated belief would warrant. The economic nationalist who argues for domestic production but who buys imported goods when the price difference matters to them personally. The health-conscious person who insists that all food should be organic but who does not apply this standard consistently when the cost is high. In each case, the behavior in private reveals something about the actual weight of the belief that the public expression conceals.

The social function of crony beliefs

Crony beliefs do not exist in isolation. They exist as part of a social ecosystem in which the expression of certain beliefs signals membership in specific communities, earns social approval from community members, and protects the believer from the social costs of holding heterodox views. The belief is not merely a representation of how things are. It is a credential: a signal of who one is and which side one is on.

This social function is not pathological in origin. The adoption of beliefs that align with one’s community (the process that article 209 described through Henrich’s account of cultural learning) is a generally reliable mechanism for acquiring a large body of approximately correct beliefs efficiently. The problem arises when the social function of a belief decouples from its epistemic function: when the social reward for expressing a belief becomes independent of, or actively in tension with, the question of whether the belief is true. In communities where the social cost of heterodoxy is high and the social reward for orthodoxy is large, the selective pressure on beliefs becomes primarily social rather than epistemic, and the belief system of the community drifts toward serving social functions rather than tracking truth.

This is the mechanism by which entire communities of intelligent, well-educated people can maintain beliefs that are inconsistent with the available evidence: not because the members of the community are stupid or dishonest, but because the social environment in which the beliefs are expressed and evaluated makes social conformity more rewarding than epistemic accuracy. The scientist who maintains a belief that is inconsistent with accumulating evidence in their field is not necessarily being dishonest: they may be responding rationally to a social environment in which the costs of abandoning the belief (loss of reputation, loss of grant funding, loss of community standing) exceed the epistemic benefits of accuracy. The voter who expresses confident support for a policy that is inconsistent with their private assessments of its likely effects is not necessarily being dishonest: they may be responding rationally to a social environment in which the expression of support is a condition of continued community membership.

The rationality of the individual response does not make the collective outcome rational. A community of individually rational actors each responding to social incentives that reward conformity over accuracy can produce a collective belief system that is systematically wrong in predictable directions: wrong in exactly the directions where the social costs of accuracy are highest.

The uncomfortable self-application

The self-application of the crony belief framework is the most important and most uncomfortable part of its use. The preceding paragraphs have been written in the third person: describing what other people do, how communities develop crony beliefs, how social incentives shape collective epistemic outcomes. But the framework is not useful if it is applied only to other people’s beliefs. The question it demands is: which of our own beliefs are crony beliefs?

The answer, almost certainly, is: more than we think, and the ones we would least like to consider. The beliefs most likely to be crony beliefs are the beliefs most strongly held, most publicly expressed, most resistant to challenge, most surrounded by emotional protection, and most tightly connected to the communities whose membership we most value. These are exactly the beliefs that the diagnostic features point toward, and they are exactly the beliefs that feel, from the inside, most certainly like merit beliefs.

There is a specific application to the political beliefs that article 208 argued should be approached with genuine curiosity about what the opposing framework sees. The insight that article 208 offered (that every political tradition tracks something real) is compatible with the recognition that most political beliefs, in most people, are crony beliefs in significant part: held not primarily because the evidence supports them but because they connect the holder to a community of people with whom they share a sense of identity and purpose. This does not make political beliefs worthless. It makes them less epistemically reliable than their holders typically believe, and it makes the practice of examining their functional role (asking not “is this belief true?” but “what would happen to our social standing if we revised it?”) a more honest approach than treating the felt conviction as evidence of justified truth-tracking.

The diagnostic question is not comfortable. It is not designed to be. It is designed to surface the gap between how our beliefs feel to us (as genuine representations of how things are) and what some of them actually are: socially calibrated positions whose continuation is secured by the costs of revision rather than by the strength of the evidence. Identifying that gap, with as much honesty as one can manage, is the application of The Conscious Look to the question that matters most: not what other people believe and why, but what we believe and why.

The asymmetry: why crony beliefs are not equally distributed

It would be convenient if crony beliefs were randomly distributed across the ideological and political spectrum, if the mechanism affected everyone equally and produced no systematic directional bias. The evidence suggests otherwise. Crony beliefs are more likely to form and persist in specific conditions: where the social costs of heterodoxy are high, where feedback from the environment is delayed or ambiguous, where the community of believers is insulated from contact with people who hold different views, and where the belief system has mechanisms for rendering contrary evidence non-disconfirming.

These conditions are not equally distributed. They are more prevalent in communities with high internal cohesion and low external contact: communities where the reinforcing effects of shared belief are strong and the corrective effects of exposure to disagreement are weak. They are more prevalent in domains where the feedback loop between belief and outcome is long and ambiguous (politics, economics, social policy) than in domains where feedback is rapid and clear (engineering, medicine with clear outcome metrics, competitive sports). And they are more prevalent in periods of high social polarization, when the costs of holding heterodox views within one’s community are elevated and the benefits of conformity are enhanced.

This asymmetry does not map cleanly onto any particular political position: both the left and the right, in different domains and at different historical moments, have shown the characteristic signature of crony belief systems. What it does suggest is that the conditions that produce crony beliefs are not equally distributed across all domains and communities at all times, and that the effort required to identify and examine one’s own crony beliefs varies accordingly. In a community with high orthodoxy pressure, the work is harder and the stakes of doing it are higher. This is precisely when the work is most important.

Notes

¹ Simler, K. (2016, November). Crony beliefs. Meltingasphalt.com. The company analogy is Simler’s own and is the most effective available illustration of the distinction between beliefs held for epistemic reasons and beliefs held for social and psychological reasons. The full essay develops several aspects of the framework that this article’s summary necessarily compresses, including a more detailed account of the evolutionary pressures that produced the crony belief mechanism and a discussion of the specific domains in which crony beliefs are most prevalent.

² The betting diagnostic has a precise theoretical foundation in the philosophy of probability. A person’s genuine probability estimate for a proposition (their actual credence, as distinct from their expressed confidence) is revealed by the odds at which they are willing to bet on it. If someone expresses 90 percent confidence in a claim but is unwilling to bet at odds that reflect 90 percent confidence, the betting behavior reveals that their actual credence is lower than their expressed confidence. This operationalization of genuine belief through betting is the basis of the decision-theoretic account of belief, associated with Frank Ramsey and Bruno de Finetti. The practical application of the diagnostic does not require formal betting: the thought experiment of asking whether one would bet, and at what odds, is sufficient to reveal the gap between expressed and genuine confidence.