How can you tell who is a valid expert, and who is full of B.S.?

On almost any topic of importance you can find a mix of valid experts (who are giving you reliable information) and false but confident-seeming “experts” (who are giving you misinformation). To make matters even more confusing, sometimes the fake experts even have very impressive credentials, and every once in a while, the real, genuine experts are entirely self-taught.

Here are 12 signs we look for in an expert to help us determine whether they are trustworthy. 

1. They have deep factual knowledge

Let’s start with the obvious: for most topics, a lot of factual knowledge is required before you can have genuine expertise. This means that a genuine expert will have an impressive command of the relevant (non-debated) facts on the topic of their expertise. Thankfully, it’s a lot easier to tell if an expert has a strong command of the non-debated facts than whether they are correct about more controversial claims. 

2. They communicate their confidence levels

Not all knowledge is equally well-established. Even theories that are widely accepted enjoy different levels of support from the relevant evidence. When an expert regularly pretends that all their claims are equally well-established, they demonstrate they are willing to make you believe something is certain when it isn’t.

It’s a good sign that someone treats their subject with the nuance expected from genuine expertise, when they indicate how confident they are (e.g., “It’s been shown in many high-quality studies that…”, or “My best guess is…”), and they explain limitations in the evidence they are using (e.g., “this is unfortunately based on just one study, but that is all that currently exists”)

3. They admit not knowing

Genuine experts also sometimes say that they don’t know the answer to a question, or that the answer is generally not known by anyone. This is important because every topic will have some unknowns, and no expert can know everything about a topic. Telling you when they don’t know is a sign that, when they say they do know, they actually do know.

4. They tell you to look at sources other than themselves

This might happen when an expert doesn’t know the answer to a question, or when they want to help you go beyond the answer they can give you. Genuine experts don’t seek to be seen as a sole arbiter of knowledge or authority on a topic (which can be an indication that ego, rather than truth-seeking, is a primary motivation for them), but instead encourage you to look at resources other than the ones they have produced.

5. They use logic and evidence

Anyone can use rhetorical devices like emotional appeals, no matter how wrong they are, but a well-reasoned argument that uses valid logic and strong evidence will tend to point toward truth. Or, put another way, using strong logic and strong evidence is easier to do when you’re right, whereas emotional appeals are no easier when you’re right than when you’re wrong.  

6. They cite high-quality evidence

Some evidence is much more reliable than other evidence, and those who rely on the less reliable kinds when the more reliable kinds exist probably aren’t doing the best job they can at figuring out the truth. For this reason, genuine experts cite high-quality evidence when it exists (e.g., looking at multiple randomized controlled trials for causal claims) rather than low-quality evidence (e.g., just talking about personal anecdotes), and when high-quality evidence doesn’t exist, they cite the highest quality evidence that does exist.

7. They acknowledge the consensus

Consensus views among experts are more often correct than the idiosyncratic views of just one or two experts. The consensus will not always be right, of course, but often it will be the best understanding we have available. That’s why reliable experts are transparent about the degree to which their opinion differs from the majority of experts, provide reasoned explanations for any deviations, and they are cautious not to present fringe theories as mainstream. This shows a deep engagement with the topic of their expertise and also an adherence to ethical standards of honesty and accuracy in communication.

8. They change their mind

Genuine experts will change their minds about topics within their expertise in response to evidence and arguments. It’s hard to become an expert in something without having been wrong from time-to-time.

That means that anyone claiming to be an expert who has never changed their mind probably has not found and corrected their mistakes. Relatedly, changing one’s mind in response to evidence is also a sign of the epistemic humility associated with genuine expertise.

Of course, if someone has a long history of being wrong, that is evidence against them being a genuine expert, not in favor of it. But, since everyone makes some mistakes, if they make mistakes from time to time and then note they were wrong and improve their beliefs, that is a sign that they are following the evidence where it leads rather than continuing to believe what they do regardless of the evidence.

9. They Steelman

When you ‘straw man’ an argument, you misrepresent or oversimplify someone else’s position to make it easier to attack or refute. Instead of dealing with the actual argument, you replace it with a weaker version that distorts the original point, which you then argue against. The opposite of this is called ‘steelmanning’, and it involves presenting the strongest possible version of an argument you’re objecting to, even if it’s more robust than the one originally presented. This approach aims to strengthen the opposing case in order to facilitate a more genuine and constructive debate. 

The most reliable experts will accurately present the strongest arguments made by those that disagree with them while pointing out flaws in those arguments, rather than focusing on just weak arguments from the other side or just mocking the other side (including ad hominem attacks rather than focusing on the substance of the claims of the other side). This is important because knocking down a weak argument from the other side of a debate does little to show the other side is wrong; you have to refute the strongest claims of the other side to actually show they are wrong. Additionally, demonstrating a knowledge of the strongest arguments against your own position shows a deeper level of expertise than only understanding the opposing point of view at a superficial level.

10. They clearly explain their reasons for believing

The philosopher Daniel Dennett has said: “if I can’t explain something I’m doing to a group of bright undergraduates, I don’t really understand it myself.” This sentiment is echoed by philosopher John Searle, who said “In general, I feel if you can’t say it clearly you don’t understand it yourself.” 

When communicating with non-experts, genuine experts are often able to give clear, easy-to-follow (and, ideally, checkable) explanations for why they believe what they believe – without dumbing down the points. They avoid unnecessary jargon and technical language (which sounds smart but makes their arguments very difficult for their audience to follow). Not every genuine expert is able to do this, but the ability to do this well is a sign of genuine expertise. This is important because an expert who cannot explain their ideas clearly will end up requiring you to believe them based on their authority rather than engaging with the arguments themselves. And sometimes, people claiming to be experts will hide behind technical expertise and jargon so that you won’t notice that their arguments are actually weak.

11. They have a track record

Sometimes genuine experts will have track records of predictions or successes that you can check, and this provides direct evidence of their knowledge or skill. Unfortunately, this only applies to some fields, like chess masters, martial experts who fight in tournaments, experts who make public predictions about the economy or politics, etc.

12. They use multiple lenses

The world is complex and multi-faceted, and any one simple theory is going to fail to explain a lot of what’s really going on. For this reason, genuine experts tend to look at problems from multiple frames and perspectives; they don’t act as though one way of looking at things solves all problems, or that one solution works for all problems, or that one simple theory explains everything.

So the next time you hear claims from an alleged expert on a topic that is important to you, you may want to consider: how many of these signs of expertise do they exhibit? You can use this checklist, considering if they:

1. have deep factual knowledge
2. communicate their confidence levels
3. admit not knowing
4. tell you to look at sources other than themselves
5. use logic and evidence
6. cite high-quality evidence
7. acknowledge the consensus
8. change their mind
9. steelman
10. clearly explain their reasons for believing
11. have a track record
12. use multiple lenses

And if you’re seeking to be an expert in something yourself, you may want to ask yourself: “to what extent do I exhibit these traits?”Being able to discern genuine expertise from B.S. requires good judgment. If you’d like to improve your skills at making accurate judgments, why not try our Calibrate Your Judgment tool, created in partnership with Open Philanthropy.

This piece first appeared on Clearer Thinking.org on April 16, 2024, and first appeared on my website on April 22, 2024.

Whereas with most of our beliefs, when we get strong counter-evidence, we become at least a bit less confident in those beliefs, with anchor beliefs, strong counter-evidence doesn’t budge us at all – it just bounces off. That means we either ignore the evidence (e.g., just don’t let ourselves think about it), explain it away (e.g., “I must have misunderstood what happened”), or change our mind about other facts to incorporate the counter-evidence without our anchor belief budging (e.g., “they must have been lying to me.”)

Understanding the role that Anchor Beliefs play in human psychology – and identifying your own personal Anchor Beliefs – can help you make better sense of the world around you. Additionally, such an understanding can help you search for false Anchor Beliefs – those apparently unquestionable truths that make up the foundations of some people’s worldviews despite being wrong! Challenging your own false anchors is very difficult, but the consequences may be life-changing.

This article provides an introduction to Anchor Beliefs, including an explanation of how they differ from other beliefs, what can make them so hard to change, and a list of common categories of Anchor Beliefs that can help you identify your own. We also give some proposals for how false Anchor Beliefs form, how you can identify these false beliefs in yourself, and what you can do to question them in a safe and productive way. If you care about understanding your own mind and the minds of other people, we think you may find this write-up valuable! 

How do Anchor Beliefs work? Instead of shifting with evidence, they shape how we see evidence.

With many low-stakes, shallow-rooted beliefs (such as which turn to take to get to a restaurant), our commitment to the belief tends to adjust when we get counter-evidence (for example, “this doesn’t look familiar – I wonder if I took the wrong turn back there”). This behavior looks like an approximate form of Bayesian updating. Anchor Beliefs don’t work like this. 

Anchor Beliefs almost never change, yet we still have to make sense of new information that we come across (some of which may strongly contradict our Anchor Beliefs). As mentioned, one solution is to warp the evidence that we receive such that we can fit it into our worldview AND keep our Anchor Belief intact at the same time. This is how Anchor Beliefs get their name: they are like huge, steel anchors securing boats to the ocean floor – only an enormously powerful current will be able to make them budge; any lesser current will simply swirl around the anchor. In this way, only incredibly powerful evidence can pose a threat to our Anchor Beliefs. And even then, our brains are highly adept at interpreting evidence so that our original Anchor Belief remains steadfast. 

Here’s a silly example to show how this might look in reality. Imagine that you HAVE to believe the walls of your house are blue. If you don’t, then everyone you love will reject you (or something equally catastrophic). So it’s really important that you believe the walls of your house are blue. This means you must bend the evidence that you receive so that your perception is compatible with this belief. But the walls of your house don’t look blue. How can you make sense of this? Maybe there is some strange-colored light in the house that makes the walls appear white. Or maybe there’s something wrong with your vision. Or maybe the walls are just an incredibly pale shade of blue that is very close to white. It’s not clear which explanation is correct, but it’s not worth wasting your time worrying about why blue walls would appear white.

While you may think that you wouldn’t fall for a false Anchor Belief like this, being particularly smart or logical doesn’t necessarily help you challenge these kinds of beliefs. You’re more likely to come up with smarter and more logical reasons why your Anchor Belief must be correct (regardless of whether or not it is). If you search hard enough, there is almost always a way to reinterpret the evidence so that your Anchor Belief can remain steadfast. 

Examples of common Anchor Beliefs

Some Anchor Beliefs are profound (say, about the origins of life), but many are prosaic (say, about the earth being spherical rather than flat). We all have large numbers of boring and trivial-sounding Anchor Beliefs such as:

• 1+1 = 2
• humans have teeth
• you live in [whatever country you believe you live in]

Consider for a moment how hard it would be for someone to convince you that you were wrong about any of these beliefs! And consider for a moment how INSANE things would be for your worldview if you did correctly come to believe that these beliefs were false. The ramifications would be so shocking that it is hard to comprehend the implications of being wrong.

What makes the above Anchor Beliefs “trivial” is they have a really high probability of being true, and virtually everyone concurs about them. Much more interesting and important to consider are Anchor Beliefs that may be false. False Anchor Beliefs are often acquired through social and cultural influences, though they can also come about in other ways (e.g., as defense mechanisms or by generalizing from a small number of traumatic experiences). Here are some common categories of Anchor Beliefs that could be false:

• Things that almost everyone you know is taught
• Certain religious beliefs learned in childhood
• Perceptions of ourselves (e.g., as good/bad)
• Views about one’s community
• Views about “enemy” groups
• Inferences from viscerally shocking first-hand experiences (e.g., “the world’s unsafe”)
• Beliefs your social group REQUIRES
• Claims that the reputation of your most trusted authority figures are staked on
• Beliefs that, if you stopped believing them, would leave you very confused about what to believe or what to do

The idea of an Anchor Belief is connected to (though not the same as) a number of other ideas, including:

• sacred values (social psychology)
• soldier vs. scout mindset (see Julia Galef’s book on the topic)
• shibboleths (the Bible)
• conflict vs. mistake theory (see Slate Star Codex)
• trapped priors (see Astral Codex Ten)
• belief updating (Connection Theory)
• core beliefs (CBT)
• leaving lines of retreat (see LessWrong)

What does changing an Anchor Belief involve?

While Anchor Beliefs almost never change, on rare occasions, we cut an anchor loose, and our boat suddenly lurches forward into the unknown. Jettisoning an anchor doesn’t necessarily take you where you want to go. It is scary, and it isn’t always safe. It may even lead you to abandon other, even more steadfast anchors. But abandoning an Anchor Belief is sometimes the only way to move forward. There’s no guarantee that the new beliefs you adopt will be correct. Consider your belief “the earth is round” (i.e., approximately an oblate spheroid, rather than flat like a disk). 

Imagine, for a moment, what would happen if you came to believe in a flat earth, and you traced out the consequences of that belief. What is NASA, then? And SpaceX? What is the field of astronomy, or geology, or cartography? Surely our government must know – so why are they keeping it from us? How long have they known this? How do they prevent the truth from getting out? Is the whole world involved in this conspiracy? Am I in danger if I publicly say it’s a conspiracy? Why don’t more people speak out about this? Have all of my friends and family also been misled? Is gravity real (and if so, how does it work on a disk)? What are the stars in the sky? 

It would be incredibly disorienting to work through all the implications that would follow from changing this once previously-unquestioned belief. Additionally, many of us have never checked whether the earth is actually round! Have you looked carefully at the arguments for this claim? And at the counterarguments to those arguments? Have you ever doubted (for even one minute of your entire adult life) that the earth is round? If we trust the evidence we receive from our eyes, many of us might assume that the earth is flat. So, how do we know that it is not? We learn this fact through our social world. (Unless, perhaps, you’ve lived by the ocean and had a habit of watching tall sailboats on the horizon.) 

Fortunately, the earth isn’t flat. At least, our Anchor Beliefs say so – we’ve never personally run experiments to check, nor have we closely scrutinized the arguments for and against this claim. (If you wanted to, there are simple experiments that you can run – for example, you could attach a camera to a helium-filled high altitude weather balloon and see the round horizon from the vantage point of the camera.) Notice that most scientific facts you believe are not as fundamental as “the earth is round,” which is a fact that has many important implications for our belief systems. Other scientific facts have fewer implications. For example, people were interested to learn that dinosaurs probably had feathers, but few (if any) had reasons to resist this update in their beliefs – because it didn’t challenge a core part of their worldview. In contrast, questioning a belief like “the earth is round” would throw most people’s belief systems into disarray. 

Interestingly enough, if you already believed there was a cabal secretly ruling the world and that scientists were controlled by this cabal, then the earth being round may merely be an ordinary belief rather than an Anchor Belief. In that case, switching your view from believing in a round earth to believing in a flat earth is not likely to be perspective-shattering – it’s easy to incorporate it into your worldview as just another thing the cabal has manipulated people to believe. 

What happens when our anchor beliefs are wrong?

Anchor Beliefs aren’t necessarily false. A lot of them are true. They just aren’t necessarily true. So, it’s important to distinguish between two types of Anchor Beliefs: (1) “False Anchors” and (2) Anchor Beliefs that happen to be true. False Anchors are obviously much more worrisome. However, they often aren’t easy to spot.

Remember, Anchor Beliefs feel to us (the believer) not like mere beliefs but like indisputable truths. However, it’s almost certain that we learned them from the people around us or derived them from shoddy generalizations, and they may not be obvious truths at all (that doesn’t mean they are false, just that they may not be indisputable like they seem). Our own Anchor Beliefs are like the dark matter of the self. They flow through us without detection, and they influence our actions.

It’s not that doubting these beliefs is impossible, but we tend to automatically dismiss skepticism towards our Anchor Beliefs (or entirely ignore evidence that contradicts them) so that we don’t have to face abandoning our worldview. False Anchor Beliefs can have negative consequences the way any false belief can: by causing your predictions to be out of sync with reality. What makes them worse than your average false belief is that they are hard to change, largely because so many other beliefs tend to rely on them. But remember that not all Anchor Beliefs are false (so it wouldn’t make sense to give up a belief merely because it’s an Anchor). 

Why is it that some of our beliefs become Anchor Beliefs?

In the simplest examples (such as 1+1=2), a belief can become an Anchor Belief (i.e., almost impossible to change) because we have so much valid evidence for it being true that our prior probability of it being true is almost 100%. These are Anchor Beliefs that we’re almost certainly right about – we’ll call them Steel Anchors because they provide a sturdy foundation for an accurate worldview. But what about those Anchor Beliefs we may well be wrong about? Beliefs that are foundational to us but are not the result of witnessing tons of valid evidence?

Let’s call these Anchor Beliefs Tin Anchors. We are almost completely unable to change our mind about Tin Anchors, despite the fact that they don’t provide a valid foundation for an accurate worldview. Five reasons that we may form Tin Anchor Beliefs:

1. It could be that Tin Anchors are a consequence of too many beliefs being piled on top of one fundamental belief, such that the fundamental belief can’t be questioned without challenging the whole pile of beliefs.
2. Tin Anchors might also be explained by the fact that it is better to have some model of the world than to have no model at all; without any kind of model, you fail to make predictions about the world, which is essential for understanding it. When an Anchor Belief falls, we’re often temporarily thrown into a state of confusion about what to believe, which makes predictions difficult. Tin Anchors, like real anchors, keep us stabilized.
3. Another explanation behind Tin Anchors is that people might grow up in social communities with poor epistemic standards; if everyone you trust tells you that something is true despite not having good evidence (especially if this starts in childhood and if this continues for a long time), these beliefs may become unmovable aspects of your worldview.
4. You might also find yourself forming Tin Anchors when you are in a situation where your brain predicts highly negative consequences from abandoning a belief. This makes changing our minds about that thing very painful and difficult. We will find all kinds of ways to cling to a belief if the alternative involves losing something that seems essential to our survival. Doubting one of these beliefs is like sticking your hand into a cage full of poisonous snakes: your mind wants to get away from the doubt as fast as possible (to avoid the severe predicted danger).
5. Tin Anchors sometimes occur when we generalize from shocking or traumatizing experiences. For instance, someone who is abused as a child might have an unshakable belief that they can’t trust other people not to hurt them (of course, some people truly cannot be trusted, but their brain may have overgeneralized this).

Finding your Anchor Beliefs

It may be valuable to ask yourself: “What are my own Tin Anchors?” If you want to consider what Tin Anchor Beliefs you may have, here are some questions that it might be helpful to ask yourself: 

• “What beliefs did I pick up from those around me that I can’t imagine not believing (yet many people in other social groups somehow manage not to believe)?”
• “What viscerally shocking experience might I have overgeneralized from that explains my worldview now?”
• “What might other people from another community claim my Anchor Beliefs are?”

These are pretty safe queries, as you’re very unlikely to stop believing your Tin Anchor Beliefs. And identifying one of your beliefs as a Tin Anchor doesn’t make it change, though it might be useful to know where your Anchors lie. Of course, it might be valuable (though costly) to try to change such an Anchor Belief if you want to. This might be something worth considering. 

So, how do you challenge your Anchor Beliefs?

Suppose you think that you’ve found one of your own Tin Anchors that you think has important implications for your life, and you actually want to examine whether it’s true. One strategy that may help is to try and clearly imagine the world where this Tin Anchor Belief turns out to be false. What is that world like? Can you deal with and accept that world? How would believing that you live in that world change your behavior and relationships? Can you accept those changes?

If you DO live in that world (where your Anchor Belief is false), would you want to believe you live in it, or would you rather pretend that your Anchor Belief isn’t false? If the answer is truly “yes” – you really would want to know if the belief is false, and you’re prepared to face the ramifications and consequences of losing that belief – then now you can truly start to put the belief to the test. Consider the strongest arguments on each side by, for instance, using our Clearer Thinking Belief Challenger program. Seek out evidence that might disconfirm the belief. Ask people that disagree with this belief why they disagree with it. Surround yourself with some people who don’t have that Anchor Belief for a little while. See if the belief survives these tests. 

How can you work around someone else’s false Anchor Beliefs?

Suppose you know someone who you think has a harmful, false belief. If your goal is to help them understand the world accurately, it is simplest to first identify the relevant Anchor Belief that this false belief relies on and assume that it cannot be changed. (Changing someone else’s foundational Anchor Belief will require an IMMENSE current or, in other words, powerful evidence that is impossible to ignore or misinterpret.) Consider what you’d say to this person to nudge them towards truth GIVEN their Anchor Beliefs. Too often, we try to change other people’s minds by attempting to shift 20-ton anchors. Instead, it is more effective to identify those Anchor Beliefs and then act as best you can under the assumption that you will almost certainly not be able to change them.

Is it bad to have Tin Anchors? Some of these Anchor Beliefs are bad, but others are fine. In any event, it seems like we may not have a choice – Anchor Beliefs might be part of the construction of human minds. Once we acknowledge that people have these almost entirely unquestionable beliefs, the world – and how people act in it – begins to make much more sense. There’s a reason so many boats are just stuck where they are (no matter how much paddling appears to be happening on the surface).

For more about Anchor Beliefs, see part 2 of this series.

This essay was first written on November 21, 2021, was published on Clearer Thinking on December 22, 2021, and was cross-posted here on December 24, 2021.

1. Mimicry: when our in-group promotes falsity that we copy

2. Incentives: when we predict that knowing the truth would feel bad or harm our objectives

3. Complexity: when the truth is hard to figure out

Examples:

1. Mimicry

• Some are Christians because all their friends and family are, too; some are atheists for the same reason.

• Some think that it makes sense to circumcise baby boys because the people they know think it’s healthy and normal; others think it’s bizarre because the people they know think foreskin is healthy and normal.

• Some believe it would harm Black Americans to defund police because their friends say so; others think it would help Black Americans because their friends say it.

2. Incentives

• If you make more money believing X, it’s going to be harder to stop believing it.

• If the idea of permanent death is terrifying to you, it’s going to be harder to stop believing in reincarnation.

• If it would make you feel really bad to find out you were wrong about something you posted online, your immediate reaction may be to deny being wrong (to others and to yourself) to shield yourself from the negative feelings.

Note that mimicry and incentives can blend together. Sometimes we mimic to fit in or to avoid being socially punished. But mimicry is even more basic than that: we seem to have a natural, in-built strong tendency to copy others. If all the people around us believe something, we usually will come to believe it too, without even questioning whether it might be false or even being aware that we copied the belief from others. 

If we see others all behave in a certain way, we’ll probably behave that way, too, unless we have strong reasons not to. This appears to be an evolutionary survival mechanism – it’s risky to (for example) eat plants that are different than the ones your tribe eats (they might be poisonous) or to avoid the behaviors everyone else does (those behaviors might be key to survival in some hidden way). In the wilderness, you can’t figure out how to survive from first principles (chances are you’ll die way too fast for that) – you need to mimic what has worked for centuries (some of which will be key to survival, some of which will be pointless, but evolutionary pressures will have weeded out most of the really harmful stuff and hung on to the most useful stuff).

3. Complexity

• It’s really not obvious how to prevent future risks from advanced artificial intelligence (though it often seems obvious to folks who’ve spent almost no time thinking about it).

• How best to prevent economic crashes is a fundamentally complicated question.

• Nobody actually seems to know how to cure cancers in general.

A weird thing about these three causes for believing falsehoods is that we are usually not aware of their effects.

1. We don’t usually realize it when we believe something just because we copied our social group.

2. We don’t usually realize when we believe something just because it would hurt us not to believe it.

3. We don’t usually realize when our analysis of a complex issue is oversimplified and misses important considerations.

Believing in falsehoods feels just like believing the truth – until the moment we genuinely face up to the possibility of being wrong.

So how do we be right more often?

1. To combat mimicry: we can keep our identities smaller (or have more of them), be more willing to be viewed as having “weird” beliefs, join social groups that value diversity of thought, learn to do less social mimicry (e.g., having greater skepticism towards in-group consensus). We can recognize that every in-group will get some things wrong (including ours) and that it’s worth investigating where ours is wrong. In summary, we can combat mimicry with social resilience and skepticism.

2. To combat harmful incentives: we can recognize that, while there can be short-term pain from accepting the truth, being truth-seeking is usually a better long-term strategy (especially because you can’t just suddenly decide to be truth-seeking when it’s convenient – it’s best to have a habit of being truth-seeking all the time). We can consider thought experiments like: “If X were true, would I rather believe it or be wrong about it?”. We can also leave “lines of retreat” so that we can decide what we’d do and how we’d move forward if we turn out to be wrong on something important. In summary, we can combat bad incentives with a Scout Mindset and a focus on seeking the truth.

3. To tackle complexity: we can use probabilistic thinking, consider multiple hypotheses, and consider the evidence for and against each one. We can train ourselves in evidence and argument evaluation, practice steel-manning arguments, talk to smart and knowledgeable people with different views and fight back against overconfidence (e.g., through calibration practice). We can also do large amounts of research when it’s important to be right. In summary, we can combat complexity with good epistemic hygiene, honed thinking skills, and self-skepticism.

This piece was first written on June 3, 2021, and first appeared on this site on November 11, 2022.

People often talk about what a problem “confirmation bias” is. But we rarely discuss what causes so many of us to search for information in a biased way.

Let’s explore some of the forces:

1. Echo chambers: our routine sources of information tend to support our worldview. Much of this is due to social ties (we tend to talk to people who are similar to us in age, geography, religion, etc.) We also trust news sources more if they share our basic ideology/assumptions. The authorities we look to will be the ones that agree with us on most of our fundamental assumptions (even if some of these assumptions could turn out to be wrong).

2. Soldier Mindset: as Julia Galef explains in her wonderful new book (the Scout Mindset), we are often not even TRYING to figure out the truth. We’re just trying to beat the other side or prove a point. In these cases, of course we have a biased search process.

3. Lack of doubt: when we’re really confident our basic premises are correct, we don’t see the need for a nuanced information search process. We’re going to go to whatever sources are convenient for filling in minor details, even if our beliefs have major unquestioned assumptions.

4. Advocacy: others are actively trying to get us to believe certain falsehoods. They do so through ads, websites, news, and other channels. For the most part, they themselves believe these falsehoods (promoting “the truth” as they see it), but occasionally it’s pure manipulation.

5. Fear of being wrong: it hurts to be wrong, especially if we’ve made the error publicly, have our identity tied up in the belief, or it challenges our understanding of the world or who to trust. We sometimes avoid finding out we are wrong the way we avoid touching a hot stove.

We don’t always seek out information in a biased way. For instance, when looking up driving directions or trying to figure out what paint to use to prevent water damage, we want the right answer, don’t have political biases, and usually have appropriate self-doubt and skepticism.

But we are liable to have a biased search for the truth when we are incentivized to have particular beliefs, such as when our social world supports just one perspective, when we’re trying to prove the other side wrong, when we have no doubt that we are right, when powerful others are devoting a lot of effort to persuade us, or when we’re too afraid of being wrong.

For more about this topic, you may want to check out my recent podcast episode with Julia Galef.

1. Soldier Altruists think it’s obvious how to improve the world and that we just need to execute those obvious steps. They see the barriers to a better world as:

(i) not enough people taking action (e.g., due to ignorance, selfishness, or propaganda), and

(ii) bad groups blocking things (e.g., corrupt politicians or greedy corporations).

2. Scout Altruists think it’s hard to figure out how to improve the world – and most attempts either don’t work, only slightly help, or make things worse. They see the barriers to a better world as:

(i) not enough understanding of causal mechanisms (e.g., due to a lack of high-quality evidence, not enough attention to the evidence we do have, not enough careful reasoning, etc.), and

(ii) too much investment in bad solutions (e.g., due to people jumping to conclusions, doing what feels good emotionally rather than what is effective, ideological blindspots, inertia, etc.)

Soldier Altruists say we need to DO and FIGHT more. Scout Altruists say we need to THINK and TEST more. Soldier Altruists are more likely to think that if we could just get people to be less selfish and more motivated to act, we would make a lot of progress towards a better world. Scout Altruists are more likely to think that if we could just get people to pay more attention to evidence and to have more good-faith debates with strong norms around the quality of argumentation, we would make a lot more progress.

Soldier Altruists may think Scout Altruists are far too reluctant to act and are wasting their time on research and debate. Scout Altruists may think Soldier Altruists are far too confident in their conclusions and are wasting their effort pushing for changes that aren’t going to help much (and which, in some cases, might even make things worse). Of course, in reality, there is a continuum between these two positions. So, on a scale from 0 (Soldier Altruist) to 10 (Scout Altruist) where do you fall? I’m probably a 7 or 8.

As some commenters have pointed out, there is a relationship between this distinction and “Conflict Theory” vs. “Mistake Theory.” I think it is related – but also distinct in important ways. Conflict theory says that there is a giant zero-sum struggle (groups fighting over fixed resources). Whereas in this case, we’re operating from a framework of altruism: “the world can be made a lot better – what’s the big barrier to that happening? Is it that we know what to do and we’re not doing it enough/with enough energy, or is it that we don’t really know what to do?”

Also, to clarify another important point brought up in the comments: I’m not asking, “do you think it’s obvious how we should improve the world if we had a magic wand that could change whatever we wanted?” – instead, the question is: “is it obvious what to do to improve the real world, given that we don’t have a magic wand?” Do we just need to put more money/time/effort/people into executing the current “obvious” strategies because they will work well if we just scale them up? Or is it pretty unclear what strategies we should even be putting more resources into (meaning that a lot of thinking, research, debate and/or evidence evaluation will typically be necessary to even figure out what is worth scaling up)?

Julia’s book (which I highly recommend): https://www.amazon.com/Scout-Mindset-Perils-Defensive-Thinking/dp/0735217556/ref=nodl_

This piece was first written on April 23rd, 2021, and first appeared on this site on January 7th, 2022.