Science is incredible, but not all studies are reliable. In fact, they frequently aren’t. So how can we tell when a study is trustworthy? Here are 14 signs to look for (for studies involving humans):
1) Large: a reasonably large number of study participants (20 per group is usually too small – the sample size should be large enough to give the study the power it needs to have a good chance of detecting an effect if the effect is indeed real)
2) Controlled: if the study is making claims to have proven that one thing causes another, then it usually will need a control group to actually demonstrate that reliably – that is, a group of participants that didn’t get whatever the treatment or intervention was. In an ideal situation, the control group gets a placebo (such as a sugar pill).
3) Randomized: if strong causal claims are being made, ideally participants would be randomized to different groups (e.g., one active treatment and one control group). This makes the two groups the same in all aspects other than the treatment, on average, and makes it much more reliable to infer causality.
4) More than statistically significant: a p-value typically captures the probability that we’d get an effect at least this extreme if there actually were no effect. For the main result, we’d ideally want it ot be well below the conventional “statistically significant” cutoff of 0.05 (not merely .048, since that hints at p-hacking).
5) Clinically significant: the size of the discovered effect should be part of the discussion (including, ideally, considerations of whether it is large enough to matter for anything practical). It’s also ideal if uncertainty in the effect size is also discussed (e.g., by providing confidence intervals on charts showing effects).
6) Simple: a statistical analysis that’s as simple as possible for that study design should be presented (even if other more complex analyses are also used). We call this the “Simplest Valid Analysis“. Complex analyses and non-standard analyses usually involve more room for fiddling to get the numbers to look good. When there is a very simple analysis of the data that would have answered the question posed, yet the researchers did only a very complicated analysis instead, that should raise questions about the robustness of the result. Furthermore, if the analysis was just performed on some subgroup (e.g., the effect was found in older men but not other groups), that should raise the question of whether they are just fishing for results. The apparent results might be just the result of noise (rolling the dice too many times and getting lucky once).
7) Replicated: ideally, other teams will find a similar result when studying a similar thing
8) Pre-registered: in the best-case scenario, study goals and analyses were planned in advance and recorded publicly before running the study. However, doing additional analyses not planned is normal (and a good thing) since the data may have led to new ideas. Still, it should be mentioned in the write-up that any such analyses were not pre-registered to avoid confusion.
9) Representative: the study population is at least reasonably similar to the group that the result will be applied to (e.g., studies on mice don’t necessarily generalize well to humans)
10) Not counter-intuitive unless it has the evidence to back it up: extremely surprising findings should require more evidence to be believed, since they are a priori less likely to be true. Of course, a lot of times the most important study findings are ones that do violate our expectations, since it’s usually less useful and interesting telling people something they already expect is true.
11) Truth-oriented: the research team seems motivated to figure out the actual truth about this topic and open-minded about the answer, rather than aiming to support an agenda. Ideally, they should attempt to prove their hypothesis wrong during the research process, not just to prove it right.
12) Caveats: limitations in the research ideally should be mentioned.
13) Materials: the study materials (e.g., exact wording of questions asked to participants) and data (when not sensitive or proprietary) should ideally be publicly released (even if after a waiting period) or, at least, available upon request from other researchers.
14) Confirmed: the original team runs a confirmatory study to double-check their main findings.
When reading a study or about one, consider the signs listed above to help you get a sense of how robust the research might be.
It’s also worth noting that the above items are what you do to help prove to others (not involved) that research is valid. In contrast, they are not necessarily exactly what you would do if you are merely trying to figure out what is true for yourself so that you can take action (e.g., make a product decision based on a study). The differences between the two cases are subtle but significant.
This piece was first written on July 2, 2020, and first appeared on my website on September 4, 2026.
Comments