Unfortunately, taking a weighted average of many different studies sometimes doesn’t work because averaging the studies can be meaningless.

Suppose a meta-analysis on “meditation for depression” tries to average the results of a one-hour app-based mindfulness meditation and a one-week silent Vipassana meditation retreat. Obviously, these can’t be averaged. But, surprisingly often, meta-analyses try things like this. There is no such thing as “the effect of meditation on depression” – since “meditation” can mean a lot of different things, and the effect size may depend a lot on what precisely is meant by “meditation” (in terms of type, dosage, etc.).

An additional problem with meta-analyses is that they sometimes combine studies of very different quality. I think low-quality studies often add more noise and bias than signal (i.e., they have a low signal-to-noise ratio). So I don’t think they should be combined with high-quality studies.

I think that for a savvy reader of papers, sometimes a better approach than meta-analyses is to sort studies by quality (highest to lowest) and skim each, trying to update one’s view on the evidence study by study, taking into account how high-quality each appears to be.

A study with a larger number of study participants tends to be more reliable. A randomized controlled trial is better than a survey in most cases. A small p-value is more reliable than a bigger one. A study that tests the exact hypothesis of interest is better than one that tests something that’s only sort of like it. And so on.

Then, you can stop when you start to get to poorly-conducted studies.

An essential aspect of this process is ensuring you understand *exactly* what was studied in each paper since an intervention’s dosage, format, and duration can matter a great deal to an outcome. It’s not enough to know it was a study of “meditation” – what kind and for how long?

Of course, many people won’t have the expertise or time to go study by study – in that case, just try to be aware of what the meta-analysis is combining together if you need to rely on it! For instance, did it eliminate low-quality studies? Are the studies being combined comparable in the format, duration, dosage, and quality of the intervention being used?

It would be great if we could treat meta-analysis as providing definite answers to scientific questions – I wish that were the case! In reality, they have a lot of problems. In addition to those already mentioned, they often suffer from publication bias (i.e., studies that found no effect were not published, and hence the meta-analysis is missing them, meaning that the set of studies being average is biased towards ones that found an effect). There are methods to check for and correct for publication bias (so you should look for those when evaluating a meta-analysis), but even those methods are imperfect.

High-quality meta-analyses also include checks for “heterogeneity” to help make sure the studies aren’t too different from each other. Even these checks have problems, though (they tend to be used in a binary way to either accept or reject the studies being “too different,” and sometimes they find the studies are not “too different” even when the studies clearly are too unrelated to be averaged).

Of course, none of this means that meta-analyses aren’t useful at times – sometimes, they are really useful. One just has to use them with caution. High-quality meta-analyses have approaches to deal with many of these issues. It’s just that, pretty often, these issues are not addressed.

If you’re interested in the topic of problems with meta-analysis, Data Colada has a great series of essays on the topic: Meaningless Means – https://datacolada.org/105 by Uri Simonsohn, Leif Nelson, and Joseph Simmons.

This piece was first written on March 10, 2023, and first appeared on this site on July 16, 2023.

Here are five ways causation without correlation can occur:

1. Averaging: increasing A sometimes causes increasing B, but other times, it causes B to decrease. The two balance out. Since correlation measures the average relationship, the correlation is zero.

For example, if you drive up a symmetrical hill and then down the other side, there’s no correlation between how many times the wheels have revolved on the hill and the car’s height above sea level, even though the revolving of the wheels causes the altitude to change.

2. Confounders: when A increases, that causes B to increase, but change is coming from C, and increasing C both causes A to increase AND causes B to decrease. The decrease in B from more C is the right amount to counteract the increase in B caused by C causing A to increase.

Example: A is in love with B and best friends with C. When C is happy, that makes A happy. When B compliments someone in front of A, that makes A unhappy. When B complements C, that makes C happy, making A happier, but the complement itself makes A unhappier, balancing it out.

3. Control: a system is designed to stabilize the value of B. Even though increases in A normally cause an increase in B, the system resists this change by exerting the opposite effect on B with the goal of keeping B constant.

For example, opening the vents would normally cool down the house, but when the heating system detects a draft, it causes the heating system to start working harder in order to keep the house at the desired 70 degrees.

4. Multiple causation: while A does cause C, C is also caused by B (so either A or B on its own is sufficient to cause C). Since A and B are both active, deactivating one has no effect on C. A and B never get deactivated at the same time, so the relationship between A and C is 0.

Example: on death row in a certain country, they administer a lethal dose of two different poisons. They have experimented with removing one or the other poison, and this did not affect whether the inmate died.

5. Deactivated causation: an increase in A causes an increase in B normally, but only if C is active. Since C is not active, there is no correlation measured between A and B. For example, force on the gas peddle only causes the car to increase speed when the car is turned on

All of this being said, while causation does not NECESSARILY imply correlation, causation USUALLY DOES imply correlation. Some software that attempts to discover causation in observational data even goes so far as to make this assumption of causation implying correlation.

This piece was first written on March 14, 2022, and first appeared on this site on December 7, 2023.