• If you’re a psychology researcher investigating possible causes of depression (Y), you may have trouble disentangling the effects of poor sleep quality (A) and anxiety (B), both of which tend to be correlated with depression.
• If you’re a health researcher investigating the causes of diabetes (Y), you may have trouble disentangling the effects of high carbohydrate intake (A) and obesity (B).
• If you’re investigating the causes of high life satisfaction (Y), you may have trouble disentangling the effects of friendship quality (A) and mental well-being (B).

In all these examples, we know that at least two of the variables (A and B) are related to the main variable (Y), but the really tricky question is to figure out what all the possible causal relationships are between the three. For instance, does A cause B, which causes Y, does Y cause both A and B, or is there some other explanation? 

In the pdf below, I sketch out 45 possible explanations to consider in situations where there are two variables that both correlate with a third variable of interest.

First of all, there are the types of causal relationships one often expects, where A and B both cause Y in simple ways (either directly or through each other):

Even if A and B really do cause Y, they could be interconnected to each other in complex ways:

It also could be the case that only A or only B causes Y, with the other variable only appearing to cause Y due to a confounding effect:

It’s also possible that Y is actually one of the causes rather than merely being caused by A and B:

Then there are situations where there is a critical other variable (or set of variables – represented as a “?” below) that are integral to the causal structure:

Finally, there are situations where Y is caused by A or B (or both), but Y also causes A or B (or both), resulting in a cyclic relationship:

Here’s a link to my pdf showing most of the possible relationships.

If you want to read about other challenges associated with untangling causality in the real world, you can read another post about this here.

I first created this diagram on April 19, 2021. I made minor edits to the diagram and wrote this piece with assistance from Clare Harris. This piece first appeared on my website on November 22, 2023.

Here are 24 different methods you could use to better understand the way humans work. In each case, I use “trying to figure out new things about post-traumatic stress disorder (PTSD)” as an example. What techniques am I leaving out here?

1. Papers: Reading academic studies (e.g., from social psychology, consumer psychology, cognitive psychology, neuroscience, etc.)

Example: reading the newest academic papers on treatments for PTSD.

2. Products: Carefully examining products and apps that have been successful (as well as, potentially, those that have failed) to understand what human needs they were or weren’t satisfying and why they succeeded or failed.

Example: looking at what mental health apps related to PTSD are popular in the app stores.

3. Outliers: Investigating extreme case studies of real-life events (e.g., case studies of people with extreme psychological conditions who may have been reported about in the news or true stories about what actually happened when people were placed into really extreme conditions).

Example: reading about cases of extreme cults and whether or not each cult produced PTSD-like symptoms in its members.

4. Society: Examining the large-scale trends and organization of societies, including potentially human cultures around the world (American culture, traditional cultures, etc.) and how they are similar or different, and which societal structures tend to be created and what the effects of these structures are, etc.

Example: investigating which sort of societies PTSD appears to be much more prevalent in.

5. Conversations: Talking to smart and insightful people about what they think is true of human psychology. You can also show them the models or theories you have so far to get their criticism and suggestions for improving them.

Example: asking really smart (non-experts) who had PTSD what their current theories about PTSD are.

6. Reading: Reading books, blog posts, media articles, etc., containing information from smart or insightful people about psychology (including potentially writing from life coaches, relationship experts, marketers, thoughtful bloggers, user experience designers, game makers, etc.)

Example: seeing what theories bloggers have proposed regarding PTSD that many academics may not have considered.

7. Experts: Reaching out to experts and talking to them to understand their models of psychology (e.g., psychologists, psychiatrists, game developers, etc.). You can also show them the models and theories you have so far to get their criticism and suggestions for improving them.

Example: talking to world-class experts in PTSD.

8. Textbooks: Reading academic textbooks about psychology, cognitive science, etc.

Example: Linehan’s textbook on DBT and thinking about possible applications of DBT in PTSD treatment.

9. Models: Looking at what models or frameworks of psychology others have created (e.g., in social psychology, theoretical cognitive neuroscience, gamification research, persuasion research, etc).

Example: looking at the model of PTSD that is used in Cognitive Behavioral Therapy treatment.

10. Intuition: Our own intuition for what’s true about humans (presumably based on our first-hand experience interacting with others, as well as our first-hand, ingrained experience of just being a human).

Example: We may have the intuition that after experiencing a traumatic event, people will tend to start having frequent disruptive thoughts about that event, even when doing unrelated things.

11. Interviews: talking to individuals or groups about their thought processes, feelings, behaviors, etc., to understand their individual psychology and then attempting to extrapolate that knowledge to humans more generally.

Example: interviewing 20 people with PTSD about what their experience of it is like and what does or does not seem to help them.

12. Application: attempting to put into practice a psychological theory (e.g., by coaching individuals using a psychological technique or by trying to build an app that implements it) and paying attention to the ways it seems to work or not work.

Example: creating a tool for people with PTSD and making it publicly available, then monitoring how people respond to it.

13. Anecdotes: psychologically surprising or interesting anecdotes that we’re confident actually happened (e.g., because they happened to us or were reported on by a trustworthy source) that may have a bearing on human psychology.

Example: a story a friend told you about how they (believe) they fully cured their own extreme PTSD in a matter of days.

14. Self-observation: careful (and honest) observation of our own thoughts/beliefs/behavior/emotions and internal processes.

Example: observing the mental processes in your own mind that seem to occur after a very upsetting event has happened.

15. Self-experimentation: simply trying techniques and paying close attention to what effects they appear to have and what it feels like to apply them.

Example: try the technique of writing down for a week every upsetting thought that you notice yourself having, as well as writing down everything you observe or learn about the experience of doing this.

16. Deduction: combining propositions we already believe to produce new propositions we didn’t realize before that may be true.

Example: knowing that women with PTSD are much more likely to have associated depression than men with PTSD, and knowing that depression is a strong risk factor for attempting suicide, we might predict that women with PTSD are more likely to attempt suicide than men with PTSD (even though they may not be more likely than men to actually commit suicide).

17. Experiments: Conduct your own online randomized controlled trials, surveys, longitudinal studies, or fMRI studies, etc.

Example: looking at which variables that don’t seem obviously PTSD related are, in fact, strong predictors of whether someone has PTSD.

18. Statistics: looking up known statistics about a phenomenon.

Example: looking up whether younger people or older people are known to be more likely to get PTSD in any given 12-month period.

19. Data: finding existing data sets (e.g., large longitudinal surveys or government data sets) and running your own statistical analyses to test hypotheses.

Example: measuring whether it is true that after someone gets a pet, they are more likely to recover from PTSD than similar people who do not get a pet. Running studies can also be a good way to search for new hypotheses. For instance, examining which of many variables in the data are most associated with a rapid recovery from PTSD.

20. Literature: looking at portrayals of psychology in literature, art, myths, stories, philosophical writings, etc.

Example: What hypotheses do Kafka’s novels give us about the nature of trauma?

21. Crowdsourcing: Request suggestions for hypotheses on a social media platform like Facebook or X or on a question-answer site like Quora.

Example: posting to Facebook asking people to suggest theories for why PTSD sometimes does and sometimes doesn’t happen after extreme trauma.

22. Social experiments: bringing a group of people together under a new set of chosen social rules or guidelines and observing what happens.

Example: getting a small group of people together with the idea that it will be a time to discuss traumas that members of the group have experienced.

23. Evolution: considering what possible evolutionary function different psychological phenomena might have.

Example: consider ways that aspects of PTSD responses might be evolutionarily adaptive.

24. Practitioners: examine how people “in the field” who benefit from being able to understand this aspect of psychology do their work.

For example, at veterans hospitals where rates of PTSD are high, investigate what techniques and approaches the therapists have developed to work successfully with their patients.

This piece was first written on November 16, 2017, and first appeared on my website on July 22, 2025.

Let’s take an example to illustrate the principle. Suppose that you have information about 1000 people selected at random from the U.S. adult population. Your dataset includes these people’s heights, weights, ages, shoe sizes, and so forth. Now, if your goal is to know the mean height of all people in America, you can produce an estimate of this quantity by averaging the heights of the 1000 people you have information about. Despite the fact that your sample contains just 1000 people, rather than the full set of 230,000,000 or so American adults of interest, your estimate will, with high probability, be within a couple of inches of the total population mean height. This is due to the fact that the 1000 people were sampled at random (so we shouldn’t expect our sample to differ from the entire population in a systematic way) and because the standard deviation of heights is not very large (if there were tremendous outliers in the data, such as 500 foot tall giants, we would need more samples to get an accurate estimate). This idea is made precise by the central limit theorem. It tells us how likely the true entire population mean is to fall different distances from our sample estimate, and says that the error of our estimate decreases like one over the square root of the size of our sample.

The same technique could work to approximate the mean weight of adult Americans, or age, or shoe size, or number of children. And in each case, the estimate would, with high probability, be quite accurate. We could even, if we liked, estimate all these quantities simultaneously if we collected all of this information about each of our 1000 people. But the more quantities we estimate, the greater the chance that at least one estimate is quite inaccurate. Since each estimate has some chance of being bad, if we make a sufficiently large number of estimates we should expect to get unlucky at some point and end up with one or more bad ones. So, if we aren’t just estimating mean height, but rather the mean of 50 different traits, we cannot claim that all 50 of these estimate are likely to be good. We should expect that some of them will be inaccurate, though we don’t know which ones.

This is where problems arise. Suppose that you are a researcher who is trying to find interesting differences between, say, southerners and northerners in the United States. Your dataset of 1000 adults contains 500 people from each group. What do you do? Well, it might seem reasonable to go ahead and compute the mean value of many different traits, and look at how these means differ between the two groups, to see if you can find any large differences that seem interesting. For instance, you may compute the average salary of each group, and see if they deviate from each other by a large enough amount to be deemed statistically significant. If they don’t, you can try another trait like IQ, or number of children, and repeat the process. If you try enough different traits, hopefully you’ll eventually find an intriguingly large difference between the groups.

The trouble is, we know that if you estimate a large number of quantities, some of them will be inaccurate, and so some of the apparent differences between your two groups may just be due to these inaccuracies. If you test enough traits, you will eventually find differences between the populations that look significant, even though it is just the result of chance.

In fact, even if northerners and southerners had no systematic differences between them, there would still be apparent differences that arose just from the particular sample of 1000 people you happened to have data on. For example, in your dataset, it just might happen that the northerners have lower numbers of children than southerners, even if this isn’t true for the underlying populations of all northerners and southerners. If you were to publish this finding, without making mention of the number of hypotheses you tested before finding it, it may seem that you had produced a meaningful result. In fact, the assessment of this result should take into account the number of hypotheses (e.g. northerners have smaller shoe sizes than southerners, northerners have greater salaries than southerners, etc.) that you tested before you discovered this one (and the p-values can be modified to include this information). The most significant seeming deviation between the groups found after testing 100 different hypotheses is very likely greatly inflated by chance. Whereas if you had only tested a small number of hypotheses against your data, and found a strong result, this would likely be a meaningful finding.

As a general rule, the greater the number of data points you have, the larger the number of quantities you can accurately estimate from your dataset. On a set of just 10 points, you may not even be able to get an accurate estimate of the mean value of a single trait (unless the trait had very slow standard deviation). Whereas on a dataset of a billion points, you probably could estimate dozens of quantities accurately.

Unfortunately, when you’re reading a paper, there is no way to tell how many hypotheses the researcher tested on his dataset unless he chooses to publish it. And there is a strong incentive to obscure this information. If a researcher releases the fact that he tested 20 hypotheses before finding 1 which was statistically significant, readers may discredit the result, or reviewers may reject it for publication. And if the researcher spent a lot of time and money collecting his dataset, it would feel like a waste to give up on the data just because his first five hypotheses tested on it don’t pan out. It might take a lot of restraint to not just keep testing hypothesis after hypothesis until he finds something publishable.

But even if researchers were excessively careful, that wouldn’t fully resolve the problem. When a hypothesis is confirmed by a dataset, we must consider whether it is truly a confirmation of the hypothesis being tested, or a result of the fact that 20 researchers tested 20 false hypotheses, and this one of the 20 happened to seem true by chance. That is, if enough hypotheses are tested over all, we may find a large number of false hypotheses among them that just happen to seem true.

What makes this problem more pernicious is that when a hypothesis fails to pan out, the result is often not published. This is due to the fact that hypothesis disconfirmations (e.g. “no association was found between cabbage eating and longevity”) are generally less interesting and harder to publish than confirmations (e.g. “an association was found between cabbage eating and longevity”). But since most new hypotheses in science turn out to be false, we should expect the number of negative results to be very large (except in situations where previously well validated results are being confirmed). Hence, the number of published test results will be much less than the number of total tests conducted, with test failures substantially underreported. So there is no good way to tell how many times a hypotheses failed to be confirmed by tests before one researcher finally ran one that seemed to confirm it. And if a very large number of false hypotheses are tested, but mostly just the ones that turn out to look true are published, you could end up with a field’s journals being flooded with false but seemingly verified hypotheses. In exploratory fields where almost all hypotheses are false, and where disconfirmations of a hypothesis are almost never published, you might even get into a situation where most published research findings are false.