Did That Treatment Actually Help You?

Image by Anna Shvets on Unsplash
A mistake we all make sometimes is attributing an improvement to whatever we've tried recently. For instance, we may get medicine from a doctor (or go to an acupuncturist) and feel better, so we conclude it worked. But did it actually work, or was it just chance? Here's a trick to help you decide: What matters (evidence-wise) is how likely that level of improvement would have been in that time period if the treatment works relative to how likely that improvement would have been if the treatm...
More

Demystifying p-values

Image by davisuko on Unsplash.
There is a tremendous amount of confusion around what a p-value actually is, despite their widespread use in science. Here is my attempt to explain the concept of p-values concisely and clearly (including why they are useful and what often goes wrong with them). — What's a p-value? — If you run a study, then (all else equal, aside from rare edge cases) the lower the p-value, the lower the chance that your results are due to random chance or luck. More precisely: a p-value is the probab...
More

Importance Hacking: a major (yet rarely-discussed) problem in science

Image created using the A.I. DALL·E
I first published this post on the Clearer Thinking blog on December 19, 2022, and first cross-posted it to this site on January 21, 2023. You have probably heard the phrase "replication crisis." It refers to the grim fact that, in a number of fields of science, when researchers attempt to replicate previously published studies, they fairly often don't get the same results. The magnitude of the problem depends on the field, but in psychology, it seems that something like 40% of studies i...
More

The many ways to make inferences

Photo by Nong V on Unsplash
There are a LOT of ways to make inferences. Many more, I think, than is generally realized. And they all have their weaknesses. You can make inferences using… (1) Deduction: As a consequence of the definition of X and Y, if X then Y. X applies to this case. Therefore Y. “Plato is a man, and all men are mortal; therefore Plato is mortal.” “For any number that is an integer, there exists another integer greater than that number. 1,000,000 is an integer. So there exists an in...
More

A thought experiment about what you’d be truly capable of doing (if you had no choice)

Photo by Samantha Sophia on Unsplash
Think of something you value that:A. multiple other people you know are capable of achieving, but thatB. you assume you would not be capable of achieving, even thoughC. you have never actually tried to do this thing well before. Now suppose, for a moment, that you have no choice but to do the thing. That is, everything you care about in the world will be destroyed if you do not achieve it in X months. Here, X could be 1 if it's a very small thing, or X could be 100 if it's a much larger thin...
More

Which Risks of Dying Are Worth Taking?

First, click here to figure out your chance of dying tomorrow. Is it worth taking a 1 in 100,000 chance of dying , in order to experience the novel thrill of sky diving? Is a 1 in 500,000 chance of death worth it to go bungee jumping? It's hard to know whether these risks are reasonable, because numbers like 100,000 or 500,000 feel so abstract to us. To think more clearly about these numbers, it helps to get our intuitions engaged. We can start by figuring out the daily risk of dying that...
More

Testing Too Many Hypotheses

For each dataset, there is a limit to what we can use that dataset to test. Using the standard p-value based methods of science, the more hypotheses we check against the data, the more likely it will be that some of these checks give inaccurate conclusions. And this presents a big problem for the way science is practiced. Let's take an example to illustrate the principle. Suppose that you have information about 1000 people selected at random from the U.S. adult population. Your dataset includ...
More

Surprised? Update your model.

In order to make predictions, your brain must have a model of reality. This model is necessarily much simpler than reality itself. To see why, imagine that you are about to drop a baseball from waist height. Your brain can't possibly know enough about the atoms composing that baseball and the air around it to simulate what will happen at the atomic level. And even if your brain did have accurate knowledge about the atoms, using information at such a fine a level of detail would be extremely comp...
More

(Almost) Everything is Uncertain

If you try to enumerate all of the things that you know with absolute, 100% certainty, you will find that the list is very small. You know that “something” exists. If you have mental experiences, then you know that “you” exist (though coming up with a reasonable definition for what “you” means can be remarkably tricky). If your mental experiences are varied, then you know that whatever exists creates varied mental experiences. With some cleverness, you may be able to add to this list ...
More

Finding Good Relationships Faster

For those who are interested in finding someone to date, a little probabilistic thinking can help a lot. To see how, let's take a moment to analyze the situation. Every time that you meet a new person, there is some chance that you will end up in a romantic relationship. If we want to assign a probability to this happening in any particular case, that probability will depend on the information that we know about both you and this other person. For instance, if we know that he or she is of a g...
More