What matters (evidence-wise) is how likely that level of improvement would have been in that time period if the treatment works relative to how likely that improvement would have been if the treatment is useless.

For something like tiredness, which tends to fluctuate a lot, feeling somewhat less tired than normal after two weeks may provide almost no evidence a treatment worked. But if you feel less tired than you have in 10 years, that could be strong evidence!

To give another example, if you’ve had a rash without a break for years, and the rash goes away in one day with a new cream, that is very strong evidence the cream worked. But if the rash very often comes and goes on its own, or it took six months of using the cream before it disappeared, its disappearance provides little evidence of effectiveness.

More formally, the amount of evidence an improvement gives you (in favor of the treatment working) is:

Bayes Factor = the probability that you’d see this level of improvement given that the treatment works / the probability that you’d see this level of improvement given that the treatment doesn’t work

In words, this is just “how many times more likely is it that you’d see this level of improvement during this period of time if the treatment works compared to if it doesn’t work.”

This Bayes Factor is what you multiply your prior odds by. So if, before trying the treatment, you thought there were 1 to 3 odds of it working (i.e., a 25% chance), and if you now you get a Bayes factor of 6, you should now believe there are 6*(1/3) = 2 to 1 odds that it works (i.e., a 66% chance).

While it’s rare to be able to do this calculation precisely, it’s this general way of thinking (in terms of relative likelihoods, comparing a world where the treatment works to one where it doesn’t) that’s important. I find this to be an especially helpful application of Bayes’ rule which can guide practical decision-making (e.g., whether to stick with a new treatment).

This piece was first written on April 15, 2023, and first appeared on this site on August 2, 2023.

• 5 types of water

• 5 types of cold coffee

• 5 types of pinot noir red wines (of varying prices)

• 5 types of pinot grigio white wines (of varying prices)

• 6 colas (including both diet and regular)

• 6 mystery liquids (all of these were actually fruit juices that had been dyed different non-standard colors – participants were informed that the liquids had been dyed)

At each station, people had to guess which item was which (from a list of all the options, except for the mystery liquids station where they had no information about what the options might be).

We also had people rate how much they enjoyed each beverage (on a seven-point scale from -3= “strongly dislike” to +3= “strongly like”), which led to some interesting findings.

— Main takeaways —

The main result of this experiment is that people are terrible at telling what they are drinking. Our taste perception is, I think, much worse than we generally believe it to be (and often, we rely on non-taste context cues to distinguish things without realizing it). The single best person in our group (at predicting which drink was which) was 50% accurate, with the mean accuracy in the group (taken across people, not predictions) being 29% and median accuracy being 24%.

Note: I DO think that people can learn to very accurately distinguish flavors with lots of repeated practice where they get feedback. But that’s a different question than whether random (untrained) people can, for example, tell colas apart in a blind taste test.

Some other tentative takeaways from the study are:

• You can probably find a cheap wine that you like as much or more than most expensive wines (in fact, you might even like relatively cheap wine better on average than expensive wine)

• You may well like bottled water better than New York tap water (many of our participants did). But if you decide to drink bottled waters, you may well be able to find cheap bottled waters that you like as much or more than expensive bottled waters.

• Random people don’t necessarily like pinot noir red wines (the average rating of how much people liked them in our group was slightly worse than neutral and substantially worse than they rated the waters)

You can see the data table of results here if you like: http://bit.ly/2NxZOcV

And here’s the mini data collection app I made for the event if you’re curious http://bit.ly/tastytasting

Take these results with a grain of salt, though, because: (a) this was a very small study on a pretty homogenous population, (b) the conclusions are based on the averages (you may well be far from average on relevant dimensions), and (c) drinking room temperature liquid from a small plastic-tasting cup is not the most accurate simulation of our experience of beverages in real life.

Below are more details on what we found.

— Can people tell which item they are drinking? —

When tasting the types of water (n = 22 people), coffees (n = 14), red wine (n = 14), white wine (n = 9), and colas (n = 21), people were not meaningfully different from chance at predicting which water, which coffee, etc. they were tasting.

For the colas, people guessed correctly 21% of the time, which was a little higher than the 17% accuracy that would be expected if they were guessing at random.

For the waters, people guessed correctly 25% of the time compared to 20% if they were guessing at random.

For the cold coffees, people guessed correctly 19% of the time compared to 20% if they were guessing at random.

For the white wines, people guessed correctly 23% of the time compared to 20% if they were guessing at random.

For the red wines, people guessed correctly 19% of the time compared to 20% if they were guessing at random.

Only on the mystery liquids were people meaningfully accurate, getting 49% correct (which you might argue is still not great). Remember that these were six different fruit juices dyed weird colors, with no context given for what each liquid was (we did not even tell people that they were juices). People had little trouble guessing orange juice (83% correct) or grapefruit juice (74% correct), with pineapple (52% correct) and grape (43% correct) being somewhat more challenging. Apple juice was hard (26% correct – though probably harder than normal because we used a fancy-ish form of Fuji apple juice rather than a more standard brand). Pomegranate juice was insanely hard (4% correct). Note that these accuracies seem to me to be pretty much in line with how commonly (I’d guess) people tend to encounter these different juices.

— Is more expensive wine better tasting? —

Among white wines (pinot grigios), the second cheapest wine was the favorite, and the very cheapest wine was the second favorite! In fact, the most expensive wine was the least liked! Among red wines (pinot noirs), the second cheapest wine was also the favorite, but the very cheapest was the second LEAST favorite. The most expensive wine was the second favorite, but people still didn’t like it much on average (they rated it as slightly worse than neutral). Overall, there was no seeming correlation between price and how much people liked the wines (in fact, the correlation between average rating and price was a little bit negative, with r=-0.34, though it’s only ten wines here, so take this with a major grain of salt).

One participant mentioned that his favorite wine of the bunch was the second cheapest, so he’s decided to buy lots of that cheap wine in the future instead of more expensive bottles. I think many people could benefit from this lesson (unless they are embarrassed liking cheap wine). Another funny finding is that people didn’t actually enjoy the red wines (pinot noirs) on average. In fact, the average liking score was negative (-0.16)! Though with the small sample size, all we can conclude is that they didn’t especially likely these wines, not that they disliked them. The white wines (pinot grigios), on the other hand, people enjoyed a little bit on average (0.38 average liking, which is between “neither like nor dislike” and “somewhat like.”

Remarkably, people said they liked the water more than both colors of wine (with an average liking rating to the water of 0.78)! Though this finding wasn’t just for wine, as people didn’t like the cold coffee either on average (giving an average rating of -0.04).

— What’s the tastiest cola? —

As with the other drinks, people had little to no ability to tell which cola was which, but on average, Coke and Pepsi (average liking of 0.71 and 0.38) did better than their diet counterparts (average liking of 0.24 for Diet Coke, 0.33 for Diet Pepsi). Coke Zero did not do well (-0.05), but Wholefoods so-called “cola” did even worse (-0.10). I personally find Diet Pepsi fairly disgusting because the artificial sweetener it uses (sucralose, a.k.a. Splenda) is different than Diet Coke (which uses aspartame), and sucralose tastes very bitter to me. But I’m not in the majority on this.

— Do people like NY tap water? —

New Yorkers often claim that our tap water is amazing, but not so much in this study. It was the least well-liked of any water tested (0.18 average liking – remember that people couldn’t tell which water they were drinking, so this isn’t due to some form of bias). But tap water is also free and environmentally friendly, so it has a lot going for it. On the other hand, Crystal Geyser (which is about half the price of the other non-tap waters tested) did well, being liked second best (0.91 liking). The fancy/expensive waters (Fiji water, Evian, Smart Water) scored likings of 1.18, 0.77, and 0.86, so Fiji was liked the best overall of all waters, but this could well be a statistical fluke given the small sample size. Incidentally, I previously had a weak belief that I liked Fiji water better than other bottled waters, so I feel slightly vindicated (though I could certainly see my belief on this being wrong, given how bad I am at telling what I’m drinking according to this study).

— What about iced coffee (served at room temperature)? —

Starbucks (Bottled) Iced Coffee pretty much crushed the coffee competition (average liking of 0.50). The only other coffee that came out with a positive rating was the substantially more expensive Chameleon cold brew (0.21). Another relatively expensive cold brew (Grady’s) did the worst of the bunch (-0.43). People also didn’t like regular hot Starbucks coffee that had been refrigerated (-0.36) or Dunkin Donuts iced coffee (-0.14).

If you’re curious, here’s a full list of the beverages tested:

WATERS (n = 22 people)

• NY Tap Water ($0 per Liter)

• Crystal Geyser ($0.66 per Liter)

• Smart Water ($1.33 per Liter)

• Evian ($1.33 per Liter)

• Fiji Water ($1.33 per Liter)

COLD COFFEES (n = 14 people)

• Dunkin Donuts Iced Coffee ($1.49 per pint)

• Starbucks (Bottled) Iced Coffee ($1.83 per pint)

• Starbuck (Refrigerated) Hot Coffee ($1.96 per pint)

• Grady’s Cold Brew ($2.75 per pint after mixing)

• Chameleon Cold-Brew ($3 per pint after mixing)

COLAS (n = 21 people)

• Coke

• Pepsi

• Wholefoods Cola

• Diet Coke

• Diet Pepsi

• Coke Zero

WHITE PINOT GRIGIO WINES (n = 9 people)

• Santa Marina 2017 ($9 – cheapest)

• Mezzacorona 2017 ($11 – 2nd cheapest)

• Kris 2016 ($14 – middle price)

• Friuli Colli Orientali ($18 – 2nd most expensive)

• Santa Margherita 2017 ($30 – most expensive)

RED PINOT NOIR WINES (n = 14 people)

• Katlyn Vineyards ($7 – cheapest)

• Mirassou ($10 – 2nd cheapest)

• Bourgogne ($20 – middle price)

• Iron Horse ($30 – 2nd most expensive)

• Jayson ($65 – most expensive)

MYSTERY LIQUIDS (dyed) (n = 23 people)

• Orange juice

• Grapefruit juice

• Pineapple juice

• Grape juice

• Apple juice

• Pomegranate juice

This essay was first written on September 23rd, 2018, and first appeared on this site on November 26th, 2021.

Can you justifiably believe that you may be able to really change the world?

There’s a certain seeming absurdity in believing you can change the world. And by “change the world,” I don’t mean playing a small (though still meaningful) cumulative role in bringing about change as part of a group of many thousands of people, each contributing incrementally. I mean, causing a large and important positive change to occur (and not merely by dumb luck) that would not have occurred (or that would have occurred much more slowly) had you not put in the effort.

The irony is that just believing you can change the world makes you, I think, significantly more likely to actually succeed at it, for at least four reasons:

(1) If you believe you can do it, you’re probably much more likely to TRY AT ALL compared to if you don’t, since the idea doesn’t seem automatically stupid or doomed to fail.

(2) When trying to create change, you’re likely to encounter numerous problems and obstacles, and if you really believing you can make change, you may be less likely to give up when these problems occur (e.g., if your first, second, and third attempts at a solution don’t work). Creating significant change probably requires smashing your way through, or skillfully dodging, EVERY ONE of these obstacles. In that vein, it’s amazing to see how often successful startups had times when they looked like they were going to fail, and if the founders had given up, they probably would have.

(3) When you really believe you can do something, others are more likely to believe it too, meaning it becomes easier to persuade others to help you or join you in your mission.

(4) If you really believe you can make change, you may feel more of a moral obligation to actually try really hard to do so. For instance, if you think you can actually ease the suffering of a million people with effort, isn’t it very important that you try hard to do so? On the other hand, if it were something you’re not capable of (or you convinced yourself of that anyway), then you wouldn’t be (or feel) obligated to solve the problem.

BOOTSTRAPPING BELIEFS

Hence, believing you can change the world is what I’ll call a “bootstrapping belief” (in the sense that it is a self-starting process). Your belief in it causes that very same belief to become truer. Another example of a bootstrapping belief might be a belief that the placebo effect has a strong effect on you. The more you believe it, the more effective the placebo effect may actually be, just by virtue of you having that belief

Yet, even with its bootstrapping nature, there is still a certain absurdity in believing you can change the world.

If you believe all that, and you’re truly willing and able to put in the effort, then maybe (with some luck) you really can.

If you don’t already believe “I may be able to really change the world,” maybe you can find yourself able to rationally endorse “I may be able to really change the world” with these extra conditions added. And if you already believe you can change the world, maybe adding these conditions to your belief will make it substantially more likely to be true.

Of course, being able to “change the world” is not binary. The amount of impact you can have is a continuous variable from “not at all” to “massive,” and the probability that you can change the world (in any particular way) is a continuous variable between 0 and 1. What I’m suggesting is that, compared to the statement “I can change the world,” the statement “I can change the world [given certain conditions]” should have a higher probability of being true.