A mistake we all make sometimes is attributing an improvement to whatever we’ve tried recently. For instance, we may get medicine from a doctor (or go to an acupuncturist) and feel better, so we conclude it worked. But did it actually work, or was it just chance? Here’s a trick to help you decide:
What matters (evidence-wise) is how likely that level of improvement would have been in that time period if the treatment works relative to how likely that improvement would have been if the treatment is useless.
For something like tiredness, which tends to fluctuate a lot, feeling somewhat less tired than normal after two weeks may provide almost no evidence a treatment worked. But if you feel less tired than you have in 10 years, that could be strong evidence!
To give another example, if you’ve had a rash without a break for years, and the rash goes away in one day with a new cream, that is very strong evidence the cream worked. But if the rash very often comes and goes on its own, or it took six months of using the cream before it disappeared, its disappearance provides little evidence of effectiveness.
More formally, the amount of evidence an improvement gives you (in favor of the treatment working) is:
Bayes Factor = the probability that you’d see this level of improvement given that the treatment works / the probability that you’d see this level of improvement given that the treatment doesn’t work
In words, this is just “how many times more likely is it that you’d see this level of improvement during this period of time if the treatment works compared to if it doesn’t work.”
This Bayes Factor is what you multiply your prior odds by. So if, before trying the treatment, you thought there were 1 to 3 odds of it working (i.e., a 25% chance), and if you now you get a Bayes factor of 6, you should now believe there are 6*(1/3) = 2 to 1 odds that it works (i.e., a 66% chance).
While it’s rare to be able to do this calculation precisely, it’s this general way of thinking (in terms of relative likelihoods, comparing a world where the treatment works to one where it doesn’t) that’s important. I find this to be an especially helpful application of Bayes’ rule which can guide practical decision-making (e.g., whether to stick with a new treatment).
This piece was first written on April 15, 2023, and first appeared on this site on August 2, 2023.
Comments