— What’s a p-value? —

If you run a study, then (all else equal, aside from rare edge cases) the lower the p-value, the lower the chance that your results are due to random chance or luck.

More precisely: a p-value is the probability you’d get a result at least as extreme as what you got IF there were actually no effect (or if some other pre-specified “null hypothesis” is true).

So it’s a probability calculated based on assuming that there is no effect (or assuming that a pre-specified “null hypothesis” is true). Here the phrase “no effect” would mean, in the case of a study on a new medicine, that the medicine doesn’t do anything.

To put it in terms of coin flips: suppose you’re trying to decide if a coin is fair (i.e., if it has an equal chance of landing on heads and tails – so that’s your “null hypothesis” in this context). You flip the coin 100 times and get 60 heads. You calculate the p-value (p=0.06).

This p-value tells you there’s a 6% chance you’d get 60 or more heads OR 60 or more tails out of 100 flips if the coin were actually fair.

What makes p-values useful is that when they are high, you usually can’t rule out your effect being due to random chance or luck. And, when they are very low, random chance is (in most cases) unlikely to be the explanation for your result.

— What’s the problem with p-values? —

In social science, p<0.05 is often used as the cutoff for a “successful” result (i.e., they treat the effect as real and potentially publishable). This is an arbitrary cutoff; there’s nothing special about 0.05. The phrase “statistically significant” is defined simply to mean that p<0.05.

There are many ways that p-values get commonly misused, creating lots of problems. For instance:

• p-values often get misinterpreted as the probability that an effect is not real (recall: p-values are actually the probability of getting a result this extreme if there is no effect, which is not the same thing)

• If you see one study where the main finding’s p-value is, say, 0.05, and another study where the main finding’s p-value is, say, 0.01, it’s tempting to conclude that the finding of the 2nd study is much less likely to be the result of chance (e.g., 1/5th as likely) than the 1st study’s finding. Unfortunately, we can’t draw this conclusion. The probability that a study’s finding is the result of chance is not the same as the p-value, and in fact, it can’t even be calculated just by knowing the p-value.

• Because a p-value threshold is often used for a result to be publishable (p<0.05 in social science), researchers sometimes engage in fishy methods to get their p-values below the threshold. This is known as “p-hacking.:

• A result’s p-value (or “statistical significance”) is sometimes focused on instead of focusing on other factors that are also important. For instance, a result may have a low p-value but be such a weak effect that it’s totally useless or uninteresting.

• While a low p-value helps you rule out the possibility that your effect is merely due to random chance, unfortunately, that’s all it helps you with. But researchers sometimes act as though it tells them more than that. Even an extremely low p-value doesn’t mean an effect is “real” or that the effect means what you think. Low p-values can result from a variety of causes, including mistakes in experimental design or confounds.

Here’s another way to think about what a p-value is and isn’t that some people find helpful: a p-value does not tell you the probability that your result is due to chance. It tells you how consistent your results are with being due to chance. (I’m paraphrasing from here.) So, the lower the p-value, the less consistent your results are with them being due to chance.

It’s interesting to note that, empirically, results with lower p-values are more likely to be genuine effects (i.e., not false positives). I looked at results for 325 psychology study replications, and when the original study p-value was at most 0.01, about 72% replicated. When p>0.01, only 48% did.

Ultimately, p-values are a useful (though often abused) statistical tool.

— BONUS APPENDIX: what’s the chance of a hypothesis being “true” if p<0.05? —

One annoying thing about p-values is that they don’t answer the question we are usually interested in. Usually, we want to know something like “What’s the probability that my hypothesis is true?” or “What’s the probability that the effect of this drug is bigger than X?” but p-values don’t tell us those things.

However, we can put a different spin on p-values to get them to answer questions that are closer to what we’re really interested in. Let’s think of p-values as giving us a decision procedure (in an overly simplified world where you either “believe” in an effect or you fail to believe in it). 

Suppose you test 100 totally separate, previously unexplored hypotheses about humans, and suppose that you commit to “believe” a hypothesis is true if and only if you get p<0.05 (and otherwise, you don’t believe it).

I think it’s realistic that in a social science context, most hypotheses studied will be false since discovering novel, publishable hypotheses about humans is hard. So let’s suppose that 80% of the hypotheses you test are *not* true. 

Finally, suppose that you use a large enough number of participants in your studies so that if you are testing for the presence of a real effect, there is an 80% chance you’ll be able to find it (this 80% figure is a common recommendation for “statistical power”). 

Under these assumptions, if you test 100 hypotheses, then you will end up believing in 20 hypotheses, and 80% of those you believe will be true (with the other 20% being false positives). That means that of the results you believe in, 80% will be correct! Of course, this assumes no mistakes are made in the process of designing the experiment, running the statistics, and so on.

Here’s how the math works out if you’re curious:

• Out of the 100 hypotheses, 20 will be true, and of those, you’ll believe 16 = 0.80 * 20 (these are the true positives) and fail to believe 4 (these are the false negatives).

• Out of the 100 hypotheses, 80 will be false, and of those, you’ll believe 4 = 0.05 * 80 (these are the false positives), and you’ll reject 76 (these are the true negatives).

Of course, if the numbers here had been different, the conclusions would be different as well. For instance, imagine if you started with 2000 hypotheses, and this time, imagine that only 1% of them were true. If the power was still 80%, then:

 • Out of the 2000 hypotheses, 20 of them would be true, and of those, you’d believe 16 (0.80 * 20) of them (these are true positives) and fail to believe 4 of them (these are false negatives).

• Out of the 2000 hypotheses, 1980 would be false, and of those, you’d believe 99 (0.05*1980) of them (these are false positives), and you’d reject the other 1881 of them (these are true negatives).

• So, altogether, you’d believe 115 (16 + 99) hypotheses, of which only 16 would’ve actually been true, so of the results you believe in, less than 14% would be correct! 

From analyses like these, we can see that the probability that a specific hypothesis is true, given that we’ve found p<0.05, depends on a variety of factors, including the sample size, the true effect size, the base rate probability that a new hypothesis tested by that researcher is true, the probability of errors being made in the experimental design or statistical analysis, and so on.

In real life:

(1) Studies often don’t use large enough numbers of participants (and so are underpowered).

(2) Researchers sometimes engage in p-hacking to artificially lower their p-values to help their papers get published.

(3) Researchers often don’t carefully track how many hypotheses they’ve really tested.

(4) The decision procedure described above is often not adhered to so strictly (e.g., a result of p=0.08 might be treated as suggestive evidence for the hypothesis, and hence the hypothesis is not rejected).

(5) Real hypotheses often have auxiliary assumptions beyond what the p-value accounts for (such as an assumption that there is a lack of confounders, a lack of serious errors in the experimental setup, and so on).

I personally don’t like thinking in terms of this decision procedure for p-values because I believe that modeling hypotheses as “true” or “false” is not a good approach to thinking clearly. This is because I believe it’s usually much better to think in terms of probabilities rather than a “true”/”false” dichotomy when trying to understand the answers to complex questions.

Some people have argued that we should switch to a Bayesian approach to hypothesis testing since such an approach avoids many of the issues of p-values (including avoiding the problematic “true”/”false” dichotomy). But it also introduces other challenges, such as how to come up with an appropriate “prior” (which represents one’s belief about the probability of the hypothesis having different strengths of effects prior to seeing the study results).

This piece was first written on December 31, 2022, and first appeared on this site on April 2, 2023.

If you read this line, please do us a favor and click here to answer one quick question.

Whereas with most of our beliefs, when we get strong counter-evidence, we become at least a bit less confident in those beliefs, with anchor beliefs, strong counter-evidence doesn’t budge us at all – it just bounces off. That means we either ignore the evidence (e.g., just don’t let ourselves think about it), explain it away (e.g., “I must have misunderstood what happened”), or change our mind about other facts to incorporate the counter-evidence without our anchor belief budging (e.g., “they must have been lying to me.”)

Understanding the role that Anchor Beliefs play in human psychology – and identifying your own personal Anchor Beliefs – can help you make better sense of the world around you. Additionally, such an understanding can help you search for false Anchor Beliefs – those apparently unquestionable truths that make up the foundations of some people’s worldviews despite being wrong! Challenging your own false anchors is very difficult, but the consequences may be life-changing.

This article provides an introduction to Anchor Beliefs, including an explanation of how they differ from other beliefs, what can make them so hard to change, and a list of common categories of Anchor Beliefs that can help you identify your own. We also give some proposals for how false Anchor Beliefs form, how you can identify these false beliefs in yourself, and what you can do to question them in a safe and productive way. If you care about understanding your own mind and the minds of other people, we think you may find this write-up valuable! 

How do Anchor Beliefs work? Instead of shifting with evidence, they shape how we see evidence.

With many low-stakes, shallow-rooted beliefs (such as which turn to take to get to a restaurant), our commitment to the belief tends to adjust when we get counter-evidence (for example, “this doesn’t look familiar – I wonder if I took the wrong turn back there”). This behavior looks like an approximate form of Bayesian updating. Anchor Beliefs don’t work like this. 

Anchor Beliefs almost never change, yet we still have to make sense of new information that we come across (some of which may strongly contradict our Anchor Beliefs). As mentioned, one solution is to warp the evidence that we receive such that we can fit it into our worldview AND keep our Anchor Belief intact at the same time. This is how Anchor Beliefs get their name: they are like huge, steel anchors securing boats to the ocean floor – only an enormously powerful current will be able to make them budge; any lesser current will simply swirl around the anchor. In this way, only incredibly powerful evidence can pose a threat to our Anchor Beliefs. And even then, our brains are highly adept at interpreting evidence so that our original Anchor Belief remains steadfast. 

Here’s a silly example to show how this might look in reality. Imagine that you HAVE to believe the walls of your house are blue. If you don’t, then everyone you love will reject you (or something equally catastrophic). So it’s really important that you believe the walls of your house are blue. This means you must bend the evidence that you receive so that your perception is compatible with this belief. But the walls of your house don’t look blue. How can you make sense of this? Maybe there is some strange-colored light in the house that makes the walls appear white. Or maybe there’s something wrong with your vision. Or maybe the walls are just an incredibly pale shade of blue that is very close to white. It’s not clear which explanation is correct, but it’s not worth wasting your time worrying about why blue walls would appear white.

While you may think that you wouldn’t fall for a false Anchor Belief like this, being particularly smart or logical doesn’t necessarily help you challenge these kinds of beliefs. You’re more likely to come up with smarter and more logical reasons why your Anchor Belief must be correct (regardless of whether or not it is). If you search hard enough, there is almost always a way to reinterpret the evidence so that your Anchor Belief can remain steadfast. 

Examples of common Anchor Beliefs

Some Anchor Beliefs are profound (say, about the origins of life), but many are prosaic (say, about the earth being spherical rather than flat). We all have large numbers of boring and trivial-sounding Anchor Beliefs such as:

• 1+1 = 2
• humans have teeth
• you live in [whatever country you believe you live in]

Consider for a moment how hard it would be for someone to convince you that you were wrong about any of these beliefs! And consider for a moment how INSANE things would be for your worldview if you did correctly come to believe that these beliefs were false. The ramifications would be so shocking that it is hard to comprehend the implications of being wrong.

What makes the above Anchor Beliefs “trivial” is they have a really high probability of being true, and virtually everyone concurs about them. Much more interesting and important to consider are Anchor Beliefs that may be false. False Anchor Beliefs are often acquired through social and cultural influences, though they can also come about in other ways (e.g., as defense mechanisms or by generalizing from a small number of traumatic experiences). Here are some common categories of Anchor Beliefs that could be false:

• Things that almost everyone you know is taught
• Certain religious beliefs learned in childhood
• Perceptions of ourselves (e.g., as good/bad)
• Views about one’s community
• Views about “enemy” groups
• Inferences from viscerally shocking first-hand experiences (e.g., “the world’s unsafe”)
• Beliefs your social group REQUIRES
• Claims that the reputation of your most trusted authority figures are staked on
• Beliefs that, if you stopped believing them, would leave you very confused about what to believe or what to do

The idea of an Anchor Belief is connected to (though not the same as) a number of other ideas, including:

• sacred values (social psychology)
• soldier vs. scout mindset (see Julia Galef’s book on the topic)
• shibboleths (the Bible)
• conflict vs. mistake theory (see Slate Star Codex)
• trapped priors (see Astral Codex Ten)
• belief updating (Connection Theory)
• core beliefs (CBT)
• leaving lines of retreat (see LessWrong)

What does changing an Anchor Belief involve?

While Anchor Beliefs almost never change, on rare occasions, we cut an anchor loose, and our boat suddenly lurches forward into the unknown. Jettisoning an anchor doesn’t necessarily take you where you want to go. It is scary, and it isn’t always safe. It may even lead you to abandon other, even more steadfast anchors. But abandoning an Anchor Belief is sometimes the only way to move forward. There’s no guarantee that the new beliefs you adopt will be correct. Consider your belief “the earth is round” (i.e., approximately an oblate spheroid, rather than flat like a disk). 

Imagine, for a moment, what would happen if you came to believe in a flat earth, and you traced out the consequences of that belief. What is NASA, then? And SpaceX? What is the field of astronomy, or geology, or cartography? Surely our government must know – so why are they keeping it from us? How long have they known this? How do they prevent the truth from getting out? Is the whole world involved in this conspiracy? Am I in danger if I publicly say it’s a conspiracy? Why don’t more people speak out about this? Have all of my friends and family also been misled? Is gravity real (and if so, how does it work on a disk)? What are the stars in the sky? 

It would be incredibly disorienting to work through all the implications that would follow from changing this once previously-unquestioned belief. Additionally, many of us have never checked whether the earth is actually round! Have you looked carefully at the arguments for this claim? And at the counterarguments to those arguments? Have you ever doubted (for even one minute of your entire adult life) that the earth is round? If we trust the evidence we receive from our eyes, many of us might assume that the earth is flat. So, how do we know that it is not? We learn this fact through our social world. (Unless, perhaps, you’ve lived by the ocean and had a habit of watching tall sailboats on the horizon.) 

Fortunately, the earth isn’t flat. At least, our Anchor Beliefs say so – we’ve never personally run experiments to check, nor have we closely scrutinized the arguments for and against this claim. (If you wanted to, there are simple experiments that you can run – for example, you could attach a camera to a helium-filled high altitude weather balloon and see the round horizon from the vantage point of the camera.) Notice that most scientific facts you believe are not as fundamental as “the earth is round,” which is a fact that has many important implications for our belief systems. Other scientific facts have fewer implications. For example, people were interested to learn that dinosaurs probably had feathers, but few (if any) had reasons to resist this update in their beliefs – because it didn’t challenge a core part of their worldview. In contrast, questioning a belief like “the earth is round” would throw most people’s belief systems into disarray. 

Interestingly enough, if you already believed there was a cabal secretly ruling the world and that scientists were controlled by this cabal, then the earth being round may merely be an ordinary belief rather than an Anchor Belief. In that case, switching your view from believing in a round earth to believing in a flat earth is not likely to be perspective-shattering – it’s easy to incorporate it into your worldview as just another thing the cabal has manipulated people to believe. 

What happens when our anchor beliefs are wrong?

Anchor Beliefs aren’t necessarily false. A lot of them are true. They just aren’t necessarily true. So, it’s important to distinguish between two types of Anchor Beliefs: (1) “False Anchors” and (2) Anchor Beliefs that happen to be true. False Anchors are obviously much more worrisome. However, they often aren’t easy to spot.

Remember, Anchor Beliefs feel to us (the believer) not like mere beliefs but like indisputable truths. However, it’s almost certain that we learned them from the people around us or derived them from shoddy generalizations, and they may not be obvious truths at all (that doesn’t mean they are false, just that they may not be indisputable like they seem). Our own Anchor Beliefs are like the dark matter of the self. They flow through us without detection, and they influence our actions.

It’s not that doubting these beliefs is impossible, but we tend to automatically dismiss skepticism towards our Anchor Beliefs (or entirely ignore evidence that contradicts them) so that we don’t have to face abandoning our worldview. False Anchor Beliefs can have negative consequences the way any false belief can: by causing your predictions to be out of sync with reality. What makes them worse than your average false belief is that they are hard to change, largely because so many other beliefs tend to rely on them. But remember that not all Anchor Beliefs are false (so it wouldn’t make sense to give up a belief merely because it’s an Anchor). 

Why is it that some of our beliefs become Anchor Beliefs?

In the simplest examples (such as 1+1=2), a belief can become an Anchor Belief (i.e., almost impossible to change) because we have so much valid evidence for it being true that our prior probability of it being true is almost 100%. These are Anchor Beliefs that we’re almost certainly right about – we’ll call them Steel Anchors because they provide a sturdy foundation for an accurate worldview. But what about those Anchor Beliefs we may well be wrong about? Beliefs that are foundational to us but are not the result of witnessing tons of valid evidence?

Let’s call these Anchor Beliefs Tin Anchors. We are almost completely unable to change our mind about Tin Anchors, despite the fact that they don’t provide a valid foundation for an accurate worldview. Five reasons that we may form Tin Anchor Beliefs:

1. It could be that Tin Anchors are a consequence of too many beliefs being piled on top of one fundamental belief, such that the fundamental belief can’t be questioned without challenging the whole pile of beliefs.
2. Tin Anchors might also be explained by the fact that it is better to have some model of the world than to have no model at all; without any kind of model, you fail to make predictions about the world, which is essential for understanding it. When an Anchor Belief falls, we’re often temporarily thrown into a state of confusion about what to believe, which makes predictions difficult. Tin Anchors, like real anchors, keep us stabilized.
3. Another explanation behind Tin Anchors is that people might grow up in social communities with poor epistemic standards; if everyone you trust tells you that something is true despite not having good evidence (especially if this starts in childhood and if this continues for a long time), these beliefs may become unmovable aspects of your worldview.
4. You might also find yourself forming Tin Anchors when you are in a situation where your brain predicts highly negative consequences from abandoning a belief. This makes changing our minds about that thing very painful and difficult. We will find all kinds of ways to cling to a belief if the alternative involves losing something that seems essential to our survival. Doubting one of these beliefs is like sticking your hand into a cage full of poisonous snakes: your mind wants to get away from the doubt as fast as possible (to avoid the severe predicted danger).
5. Tin Anchors sometimes occur when we generalize from shocking or traumatizing experiences. For instance, someone who is abused as a child might have an unshakable belief that they can’t trust other people not to hurt them (of course, some people truly cannot be trusted, but their brain may have overgeneralized this).

Finding your Anchor Beliefs

It may be valuable to ask yourself: “What are my own Tin Anchors?” If you want to consider what Tin Anchor Beliefs you may have, here are some questions that it might be helpful to ask yourself: 

• “What beliefs did I pick up from those around me that I can’t imagine not believing (yet many people in other social groups somehow manage not to believe)?”
• “What viscerally shocking experience might I have overgeneralized from that explains my worldview now?”
• “What might other people from another community claim my Anchor Beliefs are?”

These are pretty safe queries, as you’re very unlikely to stop believing your Tin Anchor Beliefs. And identifying one of your beliefs as a Tin Anchor doesn’t make it change, though it might be useful to know where your Anchors lie. Of course, it might be valuable (though costly) to try to change such an Anchor Belief if you want to. This might be something worth considering. 

So, how do you challenge your Anchor Beliefs?

Suppose you think that you’ve found one of your own Tin Anchors that you think has important implications for your life, and you actually want to examine whether it’s true. One strategy that may help is to try and clearly imagine the world where this Tin Anchor Belief turns out to be false. What is that world like? Can you deal with and accept that world? How would believing that you live in that world change your behavior and relationships? Can you accept those changes?

If you DO live in that world (where your Anchor Belief is false), would you want to believe you live in it, or would you rather pretend that your Anchor Belief isn’t false? If the answer is truly “yes” – you really would want to know if the belief is false, and you’re prepared to face the ramifications and consequences of losing that belief – then now you can truly start to put the belief to the test. Consider the strongest arguments on each side by, for instance, using our Clearer Thinking Belief Challenger program. Seek out evidence that might disconfirm the belief. Ask people that disagree with this belief why they disagree with it. Surround yourself with some people who don’t have that Anchor Belief for a little while. See if the belief survives these tests. 

How can you work around someone else’s false Anchor Beliefs?

Suppose you know someone who you think has a harmful, false belief. If your goal is to help them understand the world accurately, it is simplest to first identify the relevant Anchor Belief that this false belief relies on and assume that it cannot be changed. (Changing someone else’s foundational Anchor Belief will require an IMMENSE current or, in other words, powerful evidence that is impossible to ignore or misinterpret.) Consider what you’d say to this person to nudge them towards truth GIVEN their Anchor Beliefs. Too often, we try to change other people’s minds by attempting to shift 20-ton anchors. Instead, it is more effective to identify those Anchor Beliefs and then act as best you can under the assumption that you will almost certainly not be able to change them.

Is it bad to have Tin Anchors? Some of these Anchor Beliefs are bad, but others are fine. In any event, it seems like we may not have a choice – Anchor Beliefs might be part of the construction of human minds. Once we acknowledge that people have these almost entirely unquestionable beliefs, the world – and how people act in it – begins to make much more sense. There’s a reason so many boats are just stuck where they are (no matter how much paddling appears to be happening on the surface).

For more about Anchor Beliefs, see part 2 of this series.

This essay was first written on November 21, 2021, was published on Clearer Thinking on December 22, 2021, and was cross-posted here on December 24, 2021.

You can make inferences using…

(1) Deduction:

As a consequence of the definition of X and Y, if X then Y.

X applies to this case. Therefore Y.

“Plato is a man, and all men are mortal; therefore Plato is mortal.”

“For any number that is an integer, there exists another integer greater than that number. 1,000,000 is an integer. So there exists an integer greater than 1,000,000.”

Especially popular among philosophers and mathematicians?

Flaws: to apply to the world, you need to add in assumptions about the world, or to apply other methods of inference on top.

(2) Frequencies:

In the past, 95% of the time that X occurred, Y occurred.

X occurred. Therefore Y (with high probability).

“95% of the time when we saw a transaction identical to this one, it was fraudulent. So this transaction is fraudulent.”

Especially popular among applied statisticians and data scientists?

Flaws: You need to have a moderately large number of examples like the current one to perform calculations on, and the method assumes that those past examples were drawn from a process that is (statistically) just like the one that generated this latest example. Moreover, sometimes it is unclear what it means for “X” to have occurred. What if it’s something that’s very similar to but not quite like X that occurred – should that be counted? If we broaden our class of what counts or change to another class that still encompasses all of our prior examples, we’ll potentially get a different answer. Though, fortunately, there are plenty of cases where the class to use is fairly obvious.

(3) Models:

Given our probabilistic model of this thing, when X occurs, the probability of Y is 0.95.

X occurred. Therefore Y (with high probability).

“Given our multivariate Gaussian model of loan prices, when this loan defaults, there is a 0.95 probability of this other loan defaulting.”

Especially popular among financial engineers and risk modelers?

Flaws: hinges on the appropriateness of the parameterized probabilistic model chosen, may require a moderately large amount of past data to estimate free model parameters, and may go haywire if modeling assumptions are suddenly violated.

(4) Regression:

In prior data, as X and Z increased, the likelihood of Y increased.

X and Z are at high levels. Therefore Y.

“Height for children can be approximately predicted as an (increasing/positive) linear function of age and weight. This child is older and heavier than the others, so we predict he is also taller than the others.”

Especially common among economists and data scientists?

Flaws: often is applied with simple assumptions (e.g., linearity) that may not capture the complexity of the inference, but very large amounts of data may be needed to apply much more complex models (e.g., to use neural networks).

(5) Bayesianism:

Given my prior odds on Y being true…

And given evidence X…

And given my Bayes factor, which is my estimate of how much more likely X is to occur if Y is true than if Y is not true…

I calculate that Y is far more likely to be true than to not be true (by multiplying the prior odds by the Bayes factor to get the posterior odds).

Therefore Y (with high probability).

“My prior odds that my boss is angry at me were 1 to 4, because he’s angry at me about 20% of the time. But then he came into my office shouting and flipped over my desk, which I estimate is 200 times more likely to occur if he’s angry at me compared to if he’s not. So now the odds of him being angry at me are 200 * (1/4) = 50 to 1 in favor of him being angry.”

Not as popular as it should be?

Flaws: it is sometimes hard to know what to set your prior odds at, and it can be very hard in some cases to perform the calculation. In practice, carrying out the calculation might end up relying on subjective estimates of the odds, which can be especially tricky to guess when the evidence is not binary (i.e., not of the form “happened” vs. “didn’t happened”), or if you have lots of different pieces of evidence that are partially correlated. On the other hand, if you can do the calculations in a given instance, and have a sensible way to set a prior, this is, in my opinion, the mathematically optimal framework to use for probabilistic prediction. In that sense, we can think of many of the other approaches on this list as (hopefully pragmatic) approximations of Bayesianism (sometimes good approximations, sometimes bad ones).

(6) Theories:

Given our theory, when X occurs, Y occurs.

X occurred. Therefore Y.

“One theory is that depressed people are most at risk for suicide when they are beginning to come out of a really bad depressive episode. So as depression is remitting, patients should be carefully screened for potentially increasing risk factors.”

“When inflation rises, unemployment falls. Inflation is rising, so unemployment will fall.”

Especially popular among psychologists and economists?

Flaws: it’s very challenging to come up with reliable theories, and often you will not know how accurate such a theory is. Even if it has substantial truth to it and is often right, there may be cases where the opposite of what was predicted actually happens, and for reasons that the theory can’t explain.

(7) Causes:

We know that X causes Y to occur.

X occurred. Therefore Y.

“Rusting of gears causes increased friction, leading to greater wearing down. In this case, the gears were heavily rusted, so we expect to find a lot of wearing down.”

“This gene produces this phenotype, and we see that this gene is present, so we expect to see the phenotype.”

Especially popular among engineers and biologists?

Flaws: it’s often extremely hard to figure out causality in a highly complex system, especially in “softer” subjects like nutrition.

(8) Experts:

This expert (or prediction market, or prediction algorithm) X is 90% accurate at predicting things in this general domain of prediction.

X predicts Y. Therefore Y (with high probability).

“This prediction market has been right 90% of the time when predicting recent baseball outcomes, and in this case, they predict that the Yankees will win.”

Not as popular as it should be?

Flaws: you often don’t have access to the predictions of experts (or of prediction markets or prediction algorithms), and when you do, you usually don’t have reliable measures of their past accuracy.

(9) Metaphors:

X, which is what we are dealing with now, is metaphorically a Z.

For Z, when W is true, then obviously Y.

Now W (or its metaphorical equivalent) is true for X. Therefore Y.

“Your life is but a boat, and you are riding on the waves of your experiences. When a raging storm hits, a boat can’t be under full sail. It can’t continue at its maximum speed. You are experiencing a storm now, and so you too must learn to slow down.”

Especially popular among self-help gurus and some ancient philosophers?

Flaws: Z working as a metaphor for X doesn’t mean that all (or even most) solutions that are good for situations involving Z are appropriate (or even make any sense) for X.

(10) Similarities:

X occurred, and X is very similar to Z in properties A, B, and C.

When things similar to Z in properties A, B, and C occur, Y usually occurs.

Therefore Y (with high probability).

“This conflict is similar to the Gulf war in that…and with “Gulf”-like wars, we have always seen that…”

“This data point (with unknown label) is closest in feature space to this other data point which is labeled ‘cat,’ and all the other labeled points around that point are also labeled ‘cat,’ so this unlabeled point should also likely get the label ‘cat.’”

Especially popular among historians and within some machine learning algorithms?

Flaws: in the history case, it is difficult to know which features are the appropriate ones to use to compare similarities, and often the conclusions are based on a relatively small number of examples. In the machine learning case, a very large amount of data may be needed to train the model.

(11) Cases:

In this handful of examples (or perhaps even just one example) where X occurred, Y occurred.

X occurred. Therefore Y.

“The last time we elected a [insert political group you don’t like] as president, we saw what happened. Let’s not make that mistake again.”

“The last three times I went to action movies, I didn’t like them. So I don’t want to go to one again.”

Especially popular with politicians and with nearly everyone in daily living?

Flaws: unless we are in a situation with very little noise/variability, a few examples likely will not be enough to accurately generalize from.

(12) Intuition:

X occurred. My intuition (that I may have trouble explaining) predicts that when X occurs, Y is true. Therefore Y.

“The tone of voice he used when he talked about his family gave me a bad vibe. My feeling is that anyone who talks about their family with that tone of voice probably does not really love them.”

Popular with nearly everyone in daily living?

Flaws: our intuitions can be very well-honed in situations we’ve encountered many times and that we received feedback on (i.e., where there was some sort of answer we got about how well our intuition performed), but in highly novel situations or in situations where we receive no feedback on how well our intuition is performing, our intuitions may be highly inaccurate (even though we may not FEEL any less confident about our correctness).

This essay was first written on October 7, 2018, and first appeared on this site on December 31, 2021.