Some remarkable things I suspect are true of nearly all adults: 

1) We each hold some beliefs that are almost totally non-responsive to evidence involving some combination of our identity (who we are), our group, the nature of reality (e.g., God), or the nature of what’s good.

Examples:

• Many have an unshakable belief that they are good even as they harm the world (or believe they’re insufficient even though they’re altruistic and productive)

• Most have an unshakable belief that their in-group is good and any group opposing their group is bad

2) We assume that other people’s internal experiences are more similar to our internal experiences than they really are. Consequently, we tend to predict they’ll behave more like us than they really will.

Example:

• You’re an anxious person who avoids situations you’re afraid of, so you predict other people will be more afraid of similar situations than they really will be and that they’ll be more avoidant than they really will be

3) Emotions alter our behaviors and thoughts (increasing the likelihood of some behaviors and thoughts, decreasing the likelihood of others) in emotion-dependent ways.

Examples: 

• Physical disgust increases the chance of backing away and reduces the chance of eating soon

• Feelings of depression increase the chance of thinking thoughts about situations being hopeless or actions being pointless

4) How good or bad we feel about something happening depends on the difference between our expectations about what will happen and the reality of what actually happened.

Examples: 

• If you think someone with a gun is about to shoot, but instead, they take your wallet and run, you might feel relief (whereas normally wallet theft would be highly distressing) 

• If you expect to make $300k on a deal, you might feel bad if you “only” get $200k

5) We have multiple “drives” encoded in our brains that want different things (e.g., they have different values or goals), and these often come into conflict. Our behavior is influenced by which drives are activated and how strongly each is activated.

Examples: 

• If we smell delicious popcorn right in front of us, most of us will eat it, whereas if it’s a few feet away and we can’t smell it, we’re less likely to 

• If we’re exhausted but also slightly hungry, we may delay making food until we are more hungry or less tired

6) We are influenced by the behavioral norms and patterns demonstrated by the people around us, especially when they are people who we identify as being part of our group or similar to us.

Example: 

• If it’s common to dress or talk a certain way in a place we move to, it will increase the chance we start to dress and talk similarly

This piece was first written on October 20, 2023, and first appeared on my website on February 7, 2024.

1. Music: we prefer songs that are similar to others we like but that feel novel. If a song is too similar to what we know, then it’s derivative or boring (like listening to the same music on loop), but if it’s too novel, it is usually unappealing or dissonant. Music from other cultures can be hard for us to appreciate until we’ve listened to enough of it.

2. Learning: we learn best when an idea connects to what we already understand while also adding something on top. If it’s too far removed from what we already understand, it’s confusing, or we don’t believe it. Learning often works best when bridges are built between a person’s old understanding and some new understanding.

3. Products: we like user interfaces to be familiar, but also like them to let us do something new (or do something we value in a new, better way). An unfamiliar interface feels like a struggle to learn and can leave us frustrated. One of the great strengths of the iPhone, when it was introduced, was that the interface somehow felt familiar even though we had never used something like it before.

4. Dating: most people want those they date to be familiar in most ways (e.g., similar culture, religion, political beliefs, life goals, attitudes) yet highly distinct with respect to a few key variables. The differences being sought differ for each person, but some fairly common ones sometimes sought include: a masculine person contrasting with feminine one, a submissive person contrasting with a dominant one, a breadwinner contrasting with a homemaker, a side-kick contrasting with a leader, a joker contrasting with someone who laughs often, a strong person contrasting with someone in need of protection, or someone successful at rule-breaking contrasting with someone successful at being rule-abiding.

5. Stories: we enjoy stories that follow the standard tropes (e.g., the Hero’s Journey or the “love found → love lost → love recovered” romantic comedy), yet we want them to feel novel in some way and not too derivative. Truly novel movies are rarely as popular as ones that are executed on a classic formula but strategically deviate from it in enough key ways to feel fresh. For instance, I think The Matrix did a good job of telling a very standard “hero’s journey” story but with a number of novel-feeling elements thrown in (an intriguing setting, “bullet time” filming, etc.).

So if you’re making something, and you want people to really enjoy it, you may want to consider how you can make it Familiar Yet Different.

This piece was first written on September 18, 2020, and first appeared on this site on June 10, 2022.

Surprise is the emotion you feel when a prediction made by your brain’s model of what you are going to experience deviates substantially from what you actually experience. If you discover a large mark on your hand that you didn’t know was there, you will feel surprised. Your model of your hand did not match the visual experience produced by the light reflecting off of your hand. If you drop a baseball and it flies towards the ceiling, you will be even more surprised. Your model of what happens when you drop an object was just massively violated.

Our brain is making predictions constantly. Most of the time, these predictions are pretty accurate. When our sensory experience is in line with what our brain expects, it not only does not produce the feeling of surprise, but usually does not even push information about what was just observed into our conscious awareness. Making predictions and then checking these predictions against experience acts as a filtering mechanism. Since we can only keep a very limited amount of information in conscious awareness, it makes sense not to waste this space by filling it with boring stuff that your brain already predicted was going to happen. From the point of view of the survival of our genes, surprising experiences (i.e. ones that your brain mispredicted) are much more useful to reflect on consciously than unsurprising ones.

One thing to note about our brain’s predictions is that they do not correspond to a single, exact pattern of sensory experience. When you are in the forest, and you turn to the left, your brain cannot possibly predict the precise layout of leaves you will now see. But it does expect that you will see leaves, and that they will be (roughly speaking) within a certain range of colors, a certain range of sizes, and so forth. As long as this turns out to be true, you probably won’t even think about the fact that you are now looking at a different pattern of foliage than you were before. But if instead of trees there is a man standing next to you that you did not expect, you’ll immediately become consciously aware of that man.

By taking note of surprise when it occurs, we can learn to have truer beliefs about the world. To see how, imagine that you just heard about the Asch conformity experiments, and you find yourself surprised to learn that most people in the experiment publicly gave the wrong answer to an obvious question just because a large number of other people publicly gave this wrong answer before them. Your surprise signifies an opportunity to know something that you didn’t before, and increase the accuracy of the beliefs in your model of reality.

But how should your model be updated based on this information? A naive updating would be to add a fact to your set of beliefs such as “If 35 people incorrectly claim that one line is bigger than another, then a large percentage of people will publicly claim the same belief as the group.” The problem is that when added in this way, the fact has not truly been incorporated into you web of beliefs. It has merely been glued to the web as a dangling outlier, without altering the web itself to naturally accommodate the new information.

To usefully incorporate surprising pieces of information, ask yourself:

• Why did I find this information surprising? This can help you hone in on what parts of your model need updating.
• What prior beliefs of mine does this information contradict? Your credence in these prior beliefs should then be lowered.
• Which prior beliefs of mine does this information bolster? Your credence in these prior beliefs should be increased.
• Which beliefs of mine are most natural to tweak so that this fact is no longer surprising? Once the new information no longer seems surprising, it means it has been absorbed into your belief network.

If you find the results of the Asch experiments surprising, it may be that you underestimate what people will be willing to do to avoid looking like an outsider. Or it may be that you underestimate the extent to which people will doubt their own sensory experience when it is contradicted by the opinions of others. In either case, the information you learned from the experiment should alter what you think about things besides what will happen in that precise experimental setup. For instance, this new knowledge may update your views about how cults create agreement among their members. Or it may make you realize that if dissent is not encouraged within an organization, there is a danger that conformity will take over, and the entire group may eventually succumb to obviously wrong, but socially reinforced beliefs.

If after you have updated your model the information you have learned still seems surprising, this indicates that your beliefs need further tweaking to accommodate the new information. You should not be surprised by the same (or very similar) information twice. If a friend cancels plans on you three times in a row, and you didn’t anticipate that they may cancel that third time, you failed to sufficiently process the first two occurrences.

The next time you find yourself feeling surprised, remember that it is a valuable opportunity to update your beliefs.

Let’s hope not. Even statements that we all would say we very strongly believe, like “tomorrow the sun will rise”, and “I am not a robot” we should not assign 100% probability to. While we can be very, very, very certain that the sun will rise tomorrow and that our brains are not computers, we cannot be absolutely 100% certain. Tomorrow the sun could be destroyed by some process (perhaps a process that we don’t even understand), and there is a non-zero probability that we are the result of some extraordinarily secret and amazingly advanced government robotics projects. It is simply not reasonable to view belief as a claim of absolute certitude.

Phrases like “I believe” can also be problematic because they can imply group membership. For instance, if you say “I believe that our government should use less market regulation”, but actually you view this as a statement of identifying as a libertarian, then it may be difficult for you to engage in rational truth-seeking debate. Arguments in favor of regulation may now be processed by your brain as attacks on your in group, which means that you may feel a strong urge to deny them no matter how solid they are. And admitting that regulation is a good idea may require an adjustment to your thoughts about who you are, or a reconsideration of how reliable opinions of other members of your group are. A lot more may be at stake in the argument for you than just the truth about the facts of the case.

So what is a productive way to think about situations where our brain says to us “I believe”? Perhaps we can view this as a claim that X is likely to be true. If we take this perspective, then there are some methods that we can use to get a rough idea of the probability that we are implicitly assigning to X.

Imagine a stranger comes up to you and offers to make a bet with you. You will win one dollar from the stranger if X is in fact true, and you will pay the stranger D dollars if X turns out not to be true. We will assume for the purpose of this thought experiment that an all-knowing oracle will instantly provide you both with the correct answer.

Now, the question to ask yourself is, what is the largest value of D (the number of dollars you owe if X turns out not to be true) such that you would be willing to play this game? If you claim that a certain “I believe” statement corresponds to a 99% chance that X is true, and yet you are unwilling to pay even $20 to play this game, then your thinking has probably gone wrong. According to that 99% probability assignment, you will be taking a 1% chance of losing the $20, but will have a 99% chance of making $1, which means that the expected value  (i.e. average value) of the game is plus 79 cents (so games like this will lead you to profit a decent amount, on average). You also are only risking $20, which for many people reflects a small enough amount of money that there won’t be any noticeable life consequences for losing it. So if you are not willing to put even $20 on the line for this bet, then it is likely that one of the following things is true: (1) your estimate of there being a 99% chance of X being true does not really reflect your implicitly believed probability, (2) you are being unreasonably risk averse, or (3) $20 has a substantial amount of value to you which is why you aren’t willing to risk losing this amount.

On the flip side, suppose that you took your “I believe X” statement as only reflecting a 60% confidence that X is true. Now, the gamble is much more likely to go against you, and even if you only put up $1.20 against the other guy’s $1.00 (rather than $20 as before), the expected value of the game is only plus 12 cents .

Hence, we see that the amount you would be willing to bet is an implicit measure of how strong your belief really is (though it also necessarily will be influenced by your risk aversion). Thinking about these bets won’t yield exact belief probabilities, but they can help you determine if the implicit probability you assign to X is more like 99%, 99.999%, or 60%.

Another way to try to convert “I believe” statements into probabilistic statements is to ask yourself, “how surprised would I be if X turned out not to be true?” If the answers is, “about as surprised as I would be if I tried to guess how a spun coin would land ten times, and got it correct all 10”, then you you’ve got a probabilistic estimate of about 1 in 1000. Different levels of surprise can be thought of as roughly corresponding to probabilities.

Yet another handy trick is to ask yourself, “how often, when I believed things this strongly, was I correct in the past?” Of course, you should only count examples where you quite definitively found out the true answer afterwards. If in the past when you’ve “strongly believed” something, it turned out to be true about 90% of the time, then you now have a probabilistic estimate of sorts about future “strong beliefs”.

None of these methods is fool-proof or totally rigorous, but they can still be very useful. One of the advantages of trying to convert your “I believe X” statements to “I’m about P% certain that X is true” statements is that doing so removes some of the ego investment we may have regarding X being true. In the latter case we are openly admitting that X might turn out to be false, even if our estimate of the probability P were very well computed. What we are claiming to believe now is not X itself, but in the probability of X given the information that we are aware of. It will now be easier psychologically to face up to the truth if X turns out to be false.

Converting to rough probability estimates is also useful because it forces us to attempt to pin down what we are really claiming. If we are being ambiguous in our claim, we are more likely to realize this when we think about how much we’d be willing to bet on the claim. Ambiguity is either going to increase our uncertainty in the answer, or make the bet unverifiable.

Thinking in terms of probabilities also helps avoid issues of bias that can come about from implied group membership. If we say that something is likely, we probably won’t feel as though we have just made a claim to belong to a certain group, and others will probably not hold us to this claim as well.

A final benefit from making statement in terms of likelihood rather than belief is that it makes it easier to change our minds in front of others. If I say “I believe X”, then a person makes an argument against X, and I flip to saying “You’re right”, I may seem like I lack strong convictions or am easily persuaded or believe things for bad reasons. These are all traits that I may be judged for having. On the other hand, consider how it sounds if I say “Based on the information I have seen, I think it is probable that X is true”, then someone provides an argument against X, and I reply with “Good point, taking that information into account X doesn’t seem as likely.” In this case, I am more likely to end up sounding like a careful thinker who is updating his beliefs based on the new evidence I encounter.

When you say “I think it is quite likely that less market regulation would be good for the U.S. in terms of GDP growth” that statement is more precise, more self-reflective, and more likely to lead to productive discussion than if you just to say “I believe that our government should use less market regulation.” Converting to probabilities, even if they are only rough ones like “very likely” or “a bit better than a 50% chance” can lead to more productive discussions and less bias. So ask yourself:

• What probability do I really assign to this statement?
• How much would I bet on this?
• How surprised would I be if this turned out to be false?
• How often have I been wrong in the past when I felt this strongly?

Influences: Robin Hanson, Eliezer Yudkowsky, Divia Melwani