For instance, I just made up these stories that I’ve never heard of ever happening:

• a young child stealing their mother’s car

• a dog discovering buried treasure

And indeed, with a quick search I can confirm that these things seem to really have happened!

Though, of course, this won’t always be the case since the number of human events still pales in comparison to the number of concepts that can be mixed – for instance, I couldn’t find even one documented case of “a clown being killed by bees” (though I’m confident that at some point in history, someone was dressed in a clown suit when a bee stung them).

In any event, the preponderance of events on our planet means that something happening one single time tells us almost nothing. Having happened once is a very low bar.

And yet, to make a point in a way that people find compelling, it’s sometimes mandatory (or close to it) to give real-world examples that demonstrate the point.

This creates an awkward tension where a single real-world example often has almost no evidentiary value but has substantial persuasive power.

There are some special cases where an anecdote can provide meaningful evidence. For instance, when the anecdote is so well documented or reliable that you know it happened AND the outcome couldn’t reasonably have been caused by anything other than through the explanation the anecdote provides – such as a case study in a hospital where some experimental new treatment saves a patient with a previously incurable disease. Or when you yourself have tried something once (e.g., a self-help technique), and it seemed to work well, and that is sufficient justification for trying it again.

But in most cases, despite their usefulness in making a compelling point, anecdotes should be thought of as a way to imagine something more vividly and see more clearly specific ways it can manifest, not as evidence for something being true. They are important when explaining a concept, but usually not because they provide evidence of its validity.

This piece was first written on September 13, 2024, and first appeared on my website on October 11, 2024.

— What’s a p-value? —

If you run a study, then (all else equal, aside from rare edge cases) the lower the p-value, the lower the chance that your results are due to random chance or luck.

More precisely: a p-value is the probability you’d get a result at least as extreme as what you got IF there were actually no effect (or if some other pre-specified “null hypothesis” is true).

So it’s a probability calculated based on assuming that there is no effect (or assuming that a pre-specified “null hypothesis” is true). Here the phrase “no effect” would mean, in the case of a study on a new medicine, that the medicine doesn’t do anything.

To put it in terms of coin flips: suppose you’re trying to decide if a coin is fair (i.e., if it has an equal chance of landing on heads and tails – so that’s your “null hypothesis” in this context). You flip the coin 100 times and get 60 heads. You calculate the p-value (p=0.06).

This p-value tells you there’s a 6% chance you’d get 60 or more heads OR 60 or more tails out of 100 flips if the coin were actually fair.

What makes p-values useful is that when they are high, you usually can’t rule out your effect being due to random chance or luck. And, when they are very low, random chance is (in most cases) unlikely to be the explanation for your result.

— What’s the problem with p-values? —

In social science, p<0.05 is often used as the cutoff for a “successful” result (i.e., they treat the effect as real and potentially publishable). This is an arbitrary cutoff; there’s nothing special about 0.05. The phrase “statistically significant” is defined simply to mean that p<0.05.

There are many ways that p-values get commonly misused, creating lots of problems. For instance:

• p-values often get misinterpreted as the probability that an effect is not real (recall: p-values are actually the probability of getting a result this extreme if there is no effect, which is not the same thing)

• If you see one study where the main finding’s p-value is, say, 0.05, and another study where the main finding’s p-value is, say, 0.01, it’s tempting to conclude that the finding of the 2nd study is much less likely to be the result of chance (e.g., 1/5th as likely) than the 1st study’s finding. Unfortunately, we can’t draw this conclusion. The probability that a study’s finding is the result of chance is not the same as the p-value, and in fact, it can’t even be calculated just by knowing the p-value.

• Because a p-value threshold is often used for a result to be publishable (p<0.05 in social science), researchers sometimes engage in fishy methods to get their p-values below the threshold. This is known as “p-hacking.:

• A result’s p-value (or “statistical significance”) is sometimes focused on instead of focusing on other factors that are also important. For instance, a result may have a low p-value but be such a weak effect that it’s totally useless or uninteresting.

• While a low p-value helps you rule out the possibility that your effect is merely due to random chance, unfortunately, that’s all it helps you with. But researchers sometimes act as though it tells them more than that. Even an extremely low p-value doesn’t mean an effect is “real” or that the effect means what you think. Low p-values can result from a variety of causes, including mistakes in experimental design or confounds.

Here’s another way to think about what a p-value is and isn’t that some people find helpful: a p-value does not tell you the probability that your result is due to chance. It tells you how consistent your results are with being due to chance. (I’m paraphrasing from here.) So, the lower the p-value, the less consistent your results are with them being due to chance.

It’s interesting to note that, empirically, results with lower p-values are more likely to be genuine effects (i.e., not false positives). I looked at results for 325 psychology study replications, and when the original study p-value was at most 0.01, about 72% replicated. When p>0.01, only 48% did.

Ultimately, p-values are a useful (though often abused) statistical tool.

— BONUS APPENDIX: what’s the chance of a hypothesis being “true” if p<0.05? —

One annoying thing about p-values is that they don’t answer the question we are usually interested in. Usually, we want to know something like “What’s the probability that my hypothesis is true?” or “What’s the probability that the effect of this drug is bigger than X?” but p-values don’t tell us those things.

However, we can put a different spin on p-values to get them to answer questions that are closer to what we’re really interested in. Let’s think of p-values as giving us a decision procedure (in an overly simplified world where you either “believe” in an effect or you fail to believe in it). 

Suppose you test 100 totally separate, previously unexplored hypotheses about humans, and suppose that you commit to “believe” a hypothesis is true if and only if you get p<0.05 (and otherwise, you don’t believe it).

I think it’s realistic that in a social science context, most hypotheses studied will be false since discovering novel, publishable hypotheses about humans is hard. So let’s suppose that 80% of the hypotheses you test are *not* true. 

Finally, suppose that you use a large enough number of participants in your studies so that if you are testing for the presence of a real effect, there is an 80% chance you’ll be able to find it (this 80% figure is a common recommendation for “statistical power”). 

Under these assumptions, if you test 100 hypotheses, then you will end up believing in 20 hypotheses, and 80% of those you believe will be true (with the other 20% being false positives). That means that of the results you believe in, 80% will be correct! Of course, this assumes no mistakes are made in the process of designing the experiment, running the statistics, and so on.

Here’s how the math works out if you’re curious:

• Out of the 100 hypotheses, 20 will be true, and of those, you’ll believe 16 = 0.80 * 20 (these are the true positives) and fail to believe 4 (these are the false negatives).

• Out of the 100 hypotheses, 80 will be false, and of those, you’ll believe 4 = 0.05 * 80 (these are the false positives), and you’ll reject 76 (these are the true negatives).

Of course, if the numbers here had been different, the conclusions would be different as well. For instance, imagine if you started with 2000 hypotheses, and this time, imagine that only 1% of them were true. If the power was still 80%, then:

 • Out of the 2000 hypotheses, 20 of them would be true, and of those, you’d believe 16 (0.80 * 20) of them (these are true positives) and fail to believe 4 of them (these are false negatives).

• Out of the 2000 hypotheses, 1980 would be false, and of those, you’d believe 99 (0.05*1980) of them (these are false positives), and you’d reject the other 1881 of them (these are true negatives).

• So, altogether, you’d believe 115 (16 + 99) hypotheses, of which only 16 would’ve actually been true, so of the results you believe in, less than 14% would be correct! 

From analyses like these, we can see that the probability that a specific hypothesis is true, given that we’ve found p<0.05, depends on a variety of factors, including the sample size, the true effect size, the base rate probability that a new hypothesis tested by that researcher is true, the probability of errors being made in the experimental design or statistical analysis, and so on.

In real life:

(1) Studies often don’t use large enough numbers of participants (and so are underpowered).

(2) Researchers sometimes engage in p-hacking to artificially lower their p-values to help their papers get published.

(3) Researchers often don’t carefully track how many hypotheses they’ve really tested.

(4) The decision procedure described above is often not adhered to so strictly (e.g., a result of p=0.08 might be treated as suggestive evidence for the hypothesis, and hence the hypothesis is not rejected).

(5) Real hypotheses often have auxiliary assumptions beyond what the p-value accounts for (such as an assumption that there is a lack of confounders, a lack of serious errors in the experimental setup, and so on).

I personally don’t like thinking in terms of this decision procedure for p-values because I believe that modeling hypotheses as “true” or “false” is not a good approach to thinking clearly. This is because I believe it’s usually much better to think in terms of probabilities rather than a “true”/”false” dichotomy when trying to understand the answers to complex questions.

Some people have argued that we should switch to a Bayesian approach to hypothesis testing since such an approach avoids many of the issues of p-values (including avoiding the problematic “true”/”false” dichotomy). But it also introduces other challenges, such as how to come up with an appropriate “prior” (which represents one’s belief about the probability of the hypothesis having different strengths of effects prior to seeing the study results).

This piece was first written on December 31, 2022, and first appeared on this site on April 2, 2023.

If you read this line, please do us a favor and click here to answer one quick question.

You have probably heard the phrase “replication crisis.” It refers to the grim fact that, in a number of fields of science, when researchers attempt to replicate previously published studies, they fairly often don’t get the same results. The magnitude of the problem depends on the field, but in psychology, it seems that something like 40% of studies in top journals don’t replicate. We’ve been tackling this crisis with our new Transparent Replications project, and this post explains one of our key ideas.

Replication failures are sometimes simply due to bad luck, but more often, they are caused by p-hacking – the use of fishy statistical techniques that lead to statistically significant (but misleading or erroneous) results. As big a problem as p-hacking is, there is another substantial problem in science that gets talked about much less. Although certain subtypes of this problem have been named previously, to my knowledge, the problem itself has no name, so I’m giving it one: “Importance Hacking.”

Academics want to publish in the top journals in their field. To understand Importance Hacking, let’s consider a (slightly oversimplified) list of the three most commonly-discussed ways to get a paper published in top psychology journals:

1. Conduct valuable research – make a genuinely interesting or important discovery, or add something valuable to the state of scientific knowledge. This is, of course, what just about everyone wants to do, but it’s very, very hard!
2. Commit fraud – for instance, by making up your data. Thankfully, very few people are willing to do this because it’s so unethical. So this is by far the least used approach.
3. p-hack – use fishy statistics, HARKing (i.e., hypothesizing after the results are known), selective reporting, using hidden researcher degrees of freedom, etc., in order to get a p<0.05 result that is actually just a false positive. This is a major problem and the focus of the replication crisis. Of course, false positives can also come about without fault, due to bad luck.

But here is a fourth way to get a paper published in a top journal: Importance Hacking.

4. Importance Hack – get a result that is actually not interesting, not important, and not valuable, but write about it in such a way that reviewers are convinced it is interesting, important, and/or valuable, so that it gets published.

For research to be valuable to society (and, in an ideal world, publishable in top journals), it must be true AND interesting (or important, useful, etc.). Researchers sometimes p-hack their results to skirt around the “true” criterion (by generating interesting false positives). On the other hand, Importance Hacking is a method for skirting the “interesting” criterion.

Importance Hacking is related to concepts like hype and overselling, though hype and overselling are far more general. Importance Hacking refers specifically to a phenomenon whereby research with little to no value gets published in top journals due to the use of strategies that lead reviewers to misinterpret the work. On the other hand, hype and overselling are used in many ways in many stages of research (including to make valuable research appear even more valuable).

One way to understand importance hacking is by comparing it to p-hacking. P-hacking refers to a set of bad research practices that enable researchers to publish non-existent effects. In other words, p-hacking misleads paper reviewers into thinking that non-existent effects are real. Importance Hacking, on the other hand, encompasses a different set of bad research practices: those that lead paper reviewers to believe that real (i.e., existent) results that have little to no value actually have substantial value.

This diagram illustrates how I think Importance Hacking interferes with the pipeline of producing valuable research:

There are a number of subtypes of Importance Hacking based on the method used to make a result appear interesting/important/valuable when it’s not. Here is how I subdivide them:

Types of Importance Hacking

1. Hacking Conclusions: make it seem like you showed some interesting thing X but actually show something else (X′) which sounds similar to X but is much less interesting/important. In these cases, researchers do not truly find what they imply they have found. This phenomenon is also closely connected with validity issues.

• Example 1: showing X is true in a simple video game but claiming that X is true in real life.
• Example 2: showing A and B are correlated and claiming that A causes B (when really A and B are probably both caused by some third factor C, which makes the finding much less interesting).
• Example 3: if a researcher claims to be measuring “aggression,” and couches all conclusions in these terms but is actually measuring milliliters of hot sauce that a person puts in someone else’s food. Their result about aggression will be valid only insofar as it is true that this is a valid measure of aggression.
• Example 4: some types of hacking conclusions would fall under the terms “overclaiming” or “overgeneralizing;” Tal Yarkoni has a relevant paper called The Generalizability Crisis.

2. Hacking Novelty: refer to something in a way that makes it seem more novel or unintuitive than it is. Perhaps the result is already well known or is merely what just about everyone’s common sense would already tell them is true. In these cases, researchers really do find what they claim to have found, but what they found is not novel (despite them making it seem so). Hacking Novelty is also connected to the “Jingle-jangle” fallacy – where people can be led to believe two identical concepts are different because they have different names (or, more subtly, because they are operationalized somewhat differently).

• Example 1: showing something that is already well-known but giving it a new name that leads people to think it is something new. The concept of “grit” has received this criticism; some people claim it could turn out to be just another word for conscientiousness (or already known facets of conscientiousness) – though this question does not yet seem to be settled (different sides of this debate can be found in these papers: 1, 2, 3 and 4).
• Example 2: showing that A and B are correlated, which seems surprising given how the constructs are named, but if you were to dig into how A and B were measured, it would be obvious they would be correlated.
• Example 3: showing a common-sense result that almost everyone already would predict but making it seem like it’s not obvious (e.g., by giving it a fancy scientific name).

3. Hacking Usefulness: make a result seem useful or relevant to some important outcome when in fact, it’s useless and irrelevant. In these cases, researchers find what they claim to have found, but what they find is not useful (despite them making it sound useful).

• Example: focusing on statistical significance when the effect size is so small that the result is useless. Clinicians often distinguish between “statistical significance” and “clinical significance” to highlight the pitfalls of ignoring effect sizes when considering the importance of a finding.

4. Hacking Beauty: make a result seem clean and beautiful when in fact, it’s messy or hard to interpret. In these cases, researchers focus on certain details or results and tell a story around those, but they could have focused on other details or results that would have made the story less pretty, less clear-cut, or harder to make sense of. This is related to Giner-Sorolla’s 2012 paper Science or art: How aesthetic standards grease the way through the publication bottleneck but undermine science. Hacking beauty sometimes reduces to selective reporting of some kind (i.e., selective reporting of measures, analyses, or studies) or at least of selective focus on certain findings and not others. This becomes more difficult with pre-registration; if you have to report the results of planned analyses, there’s less room to make them look pretty (you could just say they’re pretty, but that seems like overclaiming)

• Example: emphasizing the parts of the result that tell a clean story while not including (or burying somewhere in the paper) the parts that contradict that story

Science faces multiple challenges. Over the past decade, the replication crisis and subsequent open science movement have greatly increased awareness of p-hacking as a problem. Measures have begun to be put in place to reduce p-hacking. Importance Hacking is another substantial problem, but it has received far less attention.

If a pipe is leaking from two holes and its pressure is kept fixed, then repairing one hole will result in the other one leaking faster. Similarly, as best practices increasingly become commonplace as a means to reduce p-hacking, so long as the career pressures to publish in top journals don’t let up, the occurrence of Importance Hacking may increase.

It’s time to start the conversation about how Importance Hacking can be addressed.

If you’re interested in learning more about Importance Hacking, you can listen to psychology professor Alexa Tullett and me discussing it on the Clearer Thinking podcast (there, I refer to it as “Importance Laundering,” but I now think “Importance Hacking” is a better name) or me talking about it on the Two Psychologists Four Beers podcast. We also discuss my new project, Transparent Replications, which conducts rapid replications of recently published psychology papers in top journals in an effort to shift incentives and create more reliable, replicable research. If you enjoyed this article, you may be interested in checking our replication reports and learning more about the project.

Did you like this article? If so, you may like to explore the ClearerThinking Podcast, where I have fun, in-depth conversations with brilliant people about ideas that matter. Click here to see a full list of episodes.

The oddest part is that despite a heroic attempt to do so, the study authors failed to explain why the different research teams reached such different conclusions:

“More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation.”

The hypothesis they were trying to test was whether greater immigration reduces support for social policies (such as for government-provided healthcare).

The study included data from 31 countries at up to five time periods each.

If all of the countries and time points were independent data points, that would be an effective sample size of at most 5*31 = 155.

However, the data points from one country at different points in time are highly correlated.

So, in practice, this might be equivalent to more like 75 (independent) data points.

If the (effective) sample size was equivalent to only about n=75, the 95th percentile confidence interval on a correlation could be pretty large (e.g., it could be +- 0.20), suggesting that false negatives would be very common (unless the relationship in question is pretty strong).

In view of this, I think it’s possible that this study was doomed from the start.

Why? Unless we’d expect a reasonably strong effect, maybe there just wasn’t enough data to answer the question at hand.

Some of the teams may have come to this conclusion as well. Of the 73 research teams involved in the study, one of them conducted preliminary measurement scaling tests and concluded that the hypothesis could not be reliably tested. Another team’s preregistered models failed to converge, and so that team also had no numerical results. Some teams reached more than one conclusion, and across the 89 team conclusions reached, 12 of those conclusions (13.5%) were that the hypothesis was not testable with the data provided.

Then, of course, there is the possibility of confounding variables – other factors that could be linked to both immigration and inflation that make the true relationship seem bigger or smaller than it really is.

Here’s a link to the paper if you’re interested: “Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty.”

A big thanks to Cameron Colby Thomson for pointing me to this study!

This piece was first written on November 30, 2022, and first appeared on this site on September 29, 2023.

This essay is one in a series examining sexual harassment and assault. Sexual harassment and assault occur appallingly often. We have an absurdly long way to go as a society to correct this problem.

To make progress on persistent problems, as this one clearly is, it’s often helpful to try to deeply understand the forces that drive and maintain the problem. As a small step in that direction, I ran a study surveying victims and perpetrators of sexual harassment and sexual assault, collecting quantitative and qualitative data. I’m summarizing the findings in a series of posts.

This data has helped me understand this problem a bit better, and I hope you find it does the same for you. Note that this study only scratches the surface of this very complex topic.

Warning: this post contains extensive discussion about and numerous accounts of sexual harassment and sexual assault.

I surveyed 574 people in the United States online, 52% of whom are female, using Positly.com, our participant recruitment platform. They answered multiple-choice questions about their experience (or lack thereof) as victims and/or perpetrators of sexual harassment and sexual assault, as well as a number of questions about their other characteristics. Additionally, 46% were also asked to give free-form written responses explaining some of their multiple-choice answers. The participants for this study leaned somewhat more liberal and younger than the broader United States population.

Since this is a self-report survey, people may be answering in a way that is socially desirable or otherwise obscuring the truth. That being said, the survey was 100% anonymous, and the survey takers were aware that it was anonymous, so they had nothing to lose from answering honestly. But still, it seems likely that the rates of being the victim and/or perpetrator of sexual harassment and sexual assault are underreported here.

PREVELANCE

Here are the high-level results regarding how frequently participants reported being victors and/or perpetrators:

1. Cat-calling (by/to a stranger): 32% of males and 81% of females reported being victims, 18% of males and 9% of females reported being perpetrators
2. Verbal sexual harassment: 25% of males and 62% of females reported being victims, 9% of males and 5% of females reported being perpetrators
3. Physical sexual harassment: 18% of males and 45% of females reported being victims, 4% of males and 4% of females reported being perpetrators
4. Unwanted persistent sexual advances: 33% of males and 59% of females reported being victims, 9% of males and 4% of females reported being perpetrators
5. Unwanted requests for sexual favors: 25% of males and 50% of females reported being victims, 6% of males and 4% of females reported being perpetrators
6. Unwanted sexual activity (which was continued after the perpetrator had reason to believe it was unwanted): 18% of males and 37% of females reported being victims, 8% of males and 4% of females reported being perpetrators

GENDER DIFFERENCES

Unsurprisingly, males are much more likely to be perpetrators than females, and females are much more likely to be victims than males.

We assigned each participant a score from 0 to 6 based on the number of types of sexual harassment or sexual assault they had experienced (“victim scores”), and another score from 0 to 6 based on the number of types of sexual harassment or sexual assault they had perpetrated (“perpetrator scores”).

Victim scores: females averaged 3.35 yes answers versus 1.52 for males, which means that females had about 2.2 times the level of agreement to having been the victim of different types of sexual harassment or sexual assault than males, a statistically significant result (p<0.01).

Perpetrator scores: males averaged 0.54 yes answers versus 0.30 for females, which means males had about 1.8 times more agreement to having conducted sexual harassing or sexual assaulting behaviors, a statistically significant result (p<0.01).

51% of women reported having experienced 4 or more of the 6 items in the list above as victims, vs. 19% for males.

RELATIVE FREQUENCIES OF REPORTS OF BEING A VICTIM AND PERPETRATOR

Far more people report being victims than report being perpetrators.

If we look at all participants in the study, they reported having experienced 2.4 of these types of events on average as victims, and only 0.44 of these types of events as perpetrators, meaning that people reported 5.5 times more of these events occurring from the victim perspective than the perpetrator perspective.

Another way to look at this is that while 70% of participants in the study reported having experienced 1 or more of these events as a victim, only 25% admit to having been the perpetrator in one or more of these types of events. Similarly, while 55% say they have experienced 2 or more of these types of events as victims, only 10% say they have perpetrated 2 or more. The gender breakdown is that 20% of females say they have perpetrated one or more, vs. 30% of males, and 6% of females say they have perpetrated 2 or more, vs. 14% of males.

What to make of the number of reports of being victims of these events being so dramatically smaller than the number of reports of being perpetrators of these events?

One possible interpretation is that victims are being much more honest than perpetrators, and many of the actual perpetrators are simply lying about things they’ve done. Another possibility is that perpetrators are not purposely lying, but classifying their actions differently than the victims (i.e., the victim views what happened as sexual harassment, but the perpetrator doesn’t view it as sexual harassment). It would make a self-interested kind of sense that perpetrators wouldn’t want to admit even to themselves that their past actions were bad. A third possible interpretation is that a much wider range of people are victims than are perpetrators (e.g., the repeat offenders produce many victims, raising the number of victims far above the number of perpetrators). This is consistent with other data that suggests that offending once is a strong risk factor for offending again. A fourth possibility is that victims are much more likely to readily remember their experiences of being victims than their perpetrators are to remember their experiences of being perpetrators. The reality may be a mix of these.

CONCLUSION

​So, what do we learn here? It seems that it’s common to be a victim and also common (though less so) to be a perpetrator of sexual harassment and/or assault. This is also an inherently gendered issue, with very different rates of victimization and perpetration when dividing by gender.

​
— APPENDIX —

​DEMOGRAPHICS

The demographics of the sample leaned younger than the broader U.S. population (mean age 37, median 35, with 50% of people within the range 29 to 44). The population was also more liberal than the broader U.S. (especially more socially liberal, but also more economically liberal). The median and also the most common education level of participants was a bachelor’s degree. The mean household income was $62,000, with a median of $50,000 (a little less than the U.S. household median of about $60,000). Please keep the above demographics in mind when considering the results, as they may have been different for a different population.

​—

If you’re curious exactly how the questions were asked about whether someone was a victim or perpetrator for each of the 6 categories mentioned above, here are the wordings. Each of these was asked as a “yes/no” question, and the number of yes’s on the victim questions gave each person their 0-6 victimization score, whereas the number of yes’s on the perpetrator questions gave each person their 0-6 perpetration score.

Cat-calling

1. “Have you ever been cat-called by a person you didn’t know (i.e., had a stranger make a sexual whistle, shout, or comment of a sexual nature towards you)?”
2. “Have you ever cat-called a person that you don’t know (by making a sexual whistle, shout, or comment of a sexual nature)?”

Verbal sexual harassment

1. “Have you ever had a person verbally harass you in a manner that was sexual in nature?”
2. “Have you ever verbally harassed another person in a manner that was sexual in nature?”

Physical sexual harassment

1. “Have you ever had a person physically harass you in a manner that was sexual in nature?”
2. “Have you ever physically harassed another person in a manner that was sexual in nature?”

Unwanted persistent sexual advances

1. “Have you ever had a person make unwanted sexual advances towards you that they continued to make even though they had reason to believe their sexual advances were unwanted?”
2. “Have you ever made unwanted sexual advances towards a person that you continued to make even though you believed the sexual advances were unwanted?”

Unwanted requests for sexual favors

1. “Have you ever had a person make unwanted requests for sexual favors from you that they continued to make even though they had reason to believe their requests were unwanted?”
2. “Have you ever made unwanted requests for sexual favors that you continued to make even though you believed the requests were unwanted?”

Unwanted sexual activity

1. “Have you ever had a person engage in sexual activity with you when they had reason to doubt whether you actually wanted to engage in that sexual activity, but they continued anyway?”
2. “Have you ever had sexual activity with a person when you had reason to doubt whether the person actually wanted to engage in that sexual activity, but you continued anyway?”

• If you’re a psychology researcher investigating possible causes of depression (Y), you may have trouble disentangling the effects of poor sleep quality (A) and anxiety (B), both of which tend to be correlated with depression.
• If you’re a health researcher investigating the causes of diabetes (Y), you may have trouble disentangling the effects of high carbohydrate intake (A) and obesity (B).
• If you’re investigating the causes of high life satisfaction (Y), you may have trouble disentangling the effects of friendship quality (A) and mental well-being (B).

In all these examples, we know that at least two of the variables (A and B) are related to the main variable (Y), but the really tricky question is to figure out what all the possible causal relationships are between the three. For instance, does A cause B, which causes Y, does Y cause both A and B, or is there some other explanation? 

In the pdf below, I sketch out 45 possible explanations to consider in situations where there are two variables that both correlate with a third variable of interest.

First of all, there are the types of causal relationships one often expects, where A and B both cause Y in simple ways (either directly or through each other):

Even if A and B really do cause Y, they could be interconnected to each other in complex ways:

It also could be the case that only A or only B causes Y, with the other variable only appearing to cause Y due to a confounding effect:

It’s also possible that Y is actually one of the causes rather than merely being caused by A and B:

Then there are situations where there is a critical other variable (or set of variables – represented as a “?” below) that are integral to the causal structure:

Finally, there are situations where Y is caused by A or B (or both), but Y also causes A or B (or both), resulting in a cyclic relationship:

Here’s a link to my pdf showing most of the possible relationships.

If you want to read about other challenges associated with untangling causality in the real world, you can read another post about this here.

I first created this diagram on April 19, 2021. I made minor edits to the diagram and wrote this piece with assistance from Clare Harris. This piece first appeared on my website on November 22, 2023.

1. Parkinson’s Law: Work expands so as to fill the time available for its completion.
2. Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.
3. Gates’ Law: Most people overestimate what they can do in one year and underestimate what they can do in ten years.
4. Goodhart’s Law: When a measure becomes a target, it ceases to be a good measure.
5. Hanlon’s Razor: Never attribute to malice that which is adequately explained by stupidity (or, don’t invoke conspiracy when ignorance and incompetence will suffice, as conspiracy implies intelligence).
6. Acton’s Dictum: Power tends to corrupt, and absolute power corrupts absolutely.
7. Amara’s Law: We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.
8. Benford’s Law: In a diverse collection of unrelated statistics, a given statistic has roughly a 30% chance of starting with the digit 1.
9. Betteridge’s Law: Any headline which ends in a question mark can be answered by the word ‘no’.
10. Brooks’ Law: Adding manpower to a late software project makes it later.
11. Chesterton’s Fence: Reforms should not be made until the reasoning behind the existing state of affairs is understood.
12. Claasen’s Law: Usefulness = log(technology).
13. Clarke’s First Law: When a distinguished elderly scientist states that something is possible, they are almost certainly right, but when they state something is impossible, they are probably wrong.
14. Cromwell’s Rule: Nothing but logical impossibilities have a prior probability of 0 or 1.
15. Cunningham’s Law: The best way to get the right answer on the Internet is not to ask a question, it’s to post the wrong answer.
16. Doctorow’s Law: When someone puts a lock on a thing you own, against your wishes, and doesn’t give you the key, they’re not doing it for your benefit.
17. Dunbar’s Number: Most people can’t maintain stable social relationships with more than 150 people.
18. Eroom’s Law: Drug discovery is becoming slower and more expensive over time, despite improvements in technology.
19. Gell-Mann Amnesia Effect: You’ll believe articles outside your area of expertise, even after acknowledging that neighboring articles in your area of expertise are completely wrong.
20. Gibson’s Law (or the Expert Witness Law): For each PhD (to use as an expert witness for one side) there’s an equal and opposite PhD.
21. Godwin’s Law: As an online discussion grows longer, the probability of a comparison involving Nazis or Hitler approaches one.
22. Morley-Souter’s Law (Rule 34): There is porn of it (no exceptions).
23. Greenspun’s Tenth Rule: Any sufficiently complicated C program contains an ad hoc, informally specified, bug-ridden, slow implementation of half of Common Lisp.
24. Hebb’s Law: Neurons that fire together wire together.
25. Hubble’s Law: Galaxies recede from an observer at a rate proportional to their distance to that observer.
26. Hume’s Guillotine (Is-Ought Problem): Normative statements (about what’s moral/immoral/right/wrong) cannot be deduced exclusively from descriptive statements.
27. Humphrey’s Law: Conscious attention to a task normally performed automatically can impair its performance.
28. Kranzberg’s Law: Technology is neither good nor bad; nor is it neutral.
29. Lamarck’s Principle (or “Use it or Lose it”): Use it or lose it (evolutionarily speaking, but also in the brain).
30. Lewis’s Law: The comments you’ll inevitably find on any article about feminism justify feminism.
31. Littlewood’s Law: Individuals can expect miracles to happen to them, at the rate of about one per month.
32. Maes–Garreau Law: Favorable predictions about future technology will fall at the latest possible date they can come true and still remain in the lifetime of the predictor.
33. Metcalfe’s Law: The value of a system grows as approximately the square of the number of users of the system.
34. Miller’s Law: To understand what another person is saying, you must assume that it is true and try to imagine what it could be true of.
35. Moore’s Law: Computation per dollar grows exponentially (or: number of transistors per circuit doubles roughly every 24 months).
36. Murphy’s Law: Anything that can go wrong will go wrong.
37. Alder’s Law: What cannot be settled by experiment is not worth debating.
38. O’Sullivan’s Law: All organizations that are not actually right-wing will over time become left-wing.
39. Pareto’s Principle (80/20 Rule): For many phenomena 80% of consequences stem from 20% of the causes.
40. Peter’s Principle: In a hierarchy, every employee tends to rise to his level of incompetence.
41. Poisson’s Law (or Law of Large Numbers): For independent random variables with a common distribution, the average tends to the mean as sample size increases.
42. Pournelle’s Iron Law of Bureaucracy: In bureaucracy, those devoted to the bureaucracy get control, those devoted to what it’s supposed to achieve lose influence.
43. Putt’s Law: Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand.
44. Rosenthal Effect (Pygmalion Effect): High expectations lead to an increase in performance, low expectations to a decrease in performance.
45. Schneier’s Law: Any person can invent a security system so clever that she or he can’t think of how to break it.
46. Shermer’s Law: Any sufficiently advanced extraterrestrial intelligence is indistinguishable from God.
47. Zipf’s Law: The frequency of use of the nth-most-frequently-used word in any natural language is approximately inversely proportional to n (few words are used often, most are used rarely).
48. Wirth’s Law: Software gets slower more quickly than hardware gets faster.
49. Sturgeon’s Law: Ninety percent of everything is crud.
50. Stigler’s Law: No discovery is named after its original discoverer, including this one.

This piece was first written on July 6, 2020, and first appeared on my website on May 30, 2026.

The mathematical proofs are, of course, very rarely disputed. The use of mathematics is much more often disputed.

Why do these disputes arise? I’ve observed five different types.

Disputes in Applications of Statistics to Science

(1) Disputes over philosophy:

Example 1: is it valid to assign a probability to an event that is not part of any clear sequence of events or mathematical process (e.g., trying to estimate the probability that humanity will go extinct)?

We don’t agree on what we mean by “probability.”

Example 2: under what conditions should a strange data point be considered an “outlier” and hence removed from our data before calculating our statistics?

We don’t agree on what we mean by “outlier.”

Resolving these disputes might require more consensus on what we mean by different concepts in statistics and potentially more disambiguation of possible interpretations of these concepts.

(2) Disputes over what we’re trying to achieve:

Example 1: Is a p-value a measure something that we directly care about, and if not, is it still worth using?

We don’t agree on the extent to which one of the most commonly used statistics is something we care about.

Example 2: When we’re testing multiple hypotheses at once, how should we adjust our resulting statistics to account for this “multiple hypothesis testing”?

We don’t agree on how to handle the increased rate of false positives that occurs when we put multiple statistics in a single paper.

Resolving these disputes might require more agreement on exactly what we’re trying to achieve with statistics.

(3) Disputes over what is “good enough” or “accurate enough”:

Example 1: when you’re comparing the means of two groups, when is it okay to use the very popular t-test, which assumes equal variances in the two groups, rather than the welch test, which doesn’t assume equal variances (this article does a good job of exploring what a horror show this seemingly simple question is).

We don’t agree on how much we should worry about the basic assumptions of our commonly used statistical tests being violated.

Example 2: to control for effect (i.e., “factor out” the influence of one variable on the relationship between two other variables), is it sufficient to run a regression that includes that control variable in the model?

We don’t agree on what is sufficient to remove the causal effects of one variable (that we don’t care about) on another we do care about in order to avoid having those effects contaminating our main results.

Resolving these disputes might require more agreement on how robust methods are to violations of their assumptions and clearer best practices for what to do when we expect the assumptions might be violated.

(4) Disputes due to misconceptions:

Example 1: it is often assumed that the true value has a 95% chance of falling within the 95th percentile confidence interval for that value, but that’s actually a subtle misinterpretation of the definition of a 95th percentile confidence interval.

We can easily misunderstand the extremely subtle proper interpretations of statistics.

Example 2: the trim-and-fill method is often used in meta-analysis to try to correct for the fact results that find no effect are more likely to go unpublished than results that find an effect, but unfortunately, this method sometimes doesn’t do what it is supposed.

We sometimes continue to rely on techniques out of inertia even though they are known to occasionally produce wrong results.

Resolving these disputes might require better statistical education around common misconceptions and common confusions.

(5) Disputes over what to do when we lack information:

Example 1: when we don’t have any empirical data or previous experience to use to estimate a prior distribution for a variable, how should we set our prior?

We don’t have standardized steps that everyone can agree on for all procedures.

Example 2: if we don’t have reason to believe that our data is normally distributed, but the test we would usually run requires that the distribution of the data be normal, should we instead use a non-parametric version of the test (even though it has less statistical power), or should we base whether to use the non-parametric version of the test off of the result of the p-value of a test for normality, even though doing the latter will form a “compound test” and hence potentially change the interpretation of our resulting p-value?

When certain information is lacking, we don’t have a consensus on how to adapt.

Resolving these disputes might require a more standardized agreement on dos and don’ts for situations where we lack critical information.

80/20 rule – 80% of output is caused by 20% of effort (not literally true, but true in spirit most of the time).

77/23 rule – the poorest 77% of the world’s population accounts for approximately 23% of the world’s income (GDP figures 2010) [1]

75/25 rule – not more than 25 percent of the total unlicensed seaman on board a documented vessel shall be aliens lawfully admitted to the United States for permanent residence [2]

70/30 rule – if you’re trying to persuade someone, spend 70% of the time listening and only 30% of the time talking (so as to deeply understand the other person’s perspective, goals, and incentives, and to make sure they feel deeply understood) [3]

70/20/10 rule – devote 70% of your efforts to the work that you already know how to do and are good at, 20% to other opportunities for growth that you’re confident are a good idea, and 10% to more speculative or ambitious challenges

60/40 rule – in relationships, both parties should aim to put in 60% of the work and only expect to get 40% back (or, put another way, if you think you’re doing half the work in a relationship, you’re probably doing less than half, so plan to do what feels like 60% to you) [4] [5]

50/50 rule – in conversation, you should spend about half the time listening and about half the time talking (i.e., if you’re talking far more than this, you might be hogging the conversation or annoying your conversation partner, if you’re talking far less than this you may be creating extra work or stress for your conversation partner)

40/60 rule – if you’re trying to build a brand, spend only 40% of the time creating content and 60% of the time promoting it [6]

30/70 rule – if you run an organization, you should spend 30% of the time working on leading the organization in new initiatives and 70% managing the execution of existing initiatives [7]

25/69/6 rule – 25% of the rules on this list were written by me for the purpose of this post, 69% were cooked up by others in order to sell books or make snappy headlines, 6% are regulations about seamen.

10/90 rule – 10% of what your experience is like is determined by what happens to you, 90% is determined by how you react to and think about what happens to you [8]

3/97 rule – if you spend three years working your ass off at something, you can become better than at least 97% of people in the world at it

1/99 rule – 1% of the artists (or musicians) make 99% of the money

1/9/90 rule – in online communities, 1% of users contribute most of the activity, 9% of users each contribute a little bit of activity, and 90% are lurkers who never contribute

0/100 rule – 0% of the rules on this list are 100% true, but 100% of the rules on this list are more than 0% true (except perhaps this one)

[1] http://www.statisticalforensics.com/economic-inequality-cha…
[2] https://www.uscg.mil/…/Citizenship%20Req…/75%2025%20Rule.pdf
[3] http://hansengroupcompany.com/the-7030-rule-of-communicati…/
[4] https://www.theodysseyonline.com/60-40-relationship-rule
[5] https://rogerrelevant.wordpress.com/2013/05/29/about-roger/
[6] https://www.hatchforgood.org/…/the-40-60-content-rule-less-…
[7] http://www.talkingstory.org/2009/07/the-3070-rule/
[8] https://www.slideshare.net/Pre…/the-10-90-rule-stephen-covey

Let’s take an example to illustrate the principle. Suppose that you have information about 1000 people selected at random from the U.S. adult population. Your dataset includes these people’s heights, weights, ages, shoe sizes, and so forth. Now, if your goal is to know the mean height of all people in America, you can produce an estimate of this quantity by averaging the heights of the 1000 people you have information about. Despite the fact that your sample contains just 1000 people, rather than the full set of 230,000,000 or so American adults of interest, your estimate will, with high probability, be within a couple of inches of the total population mean height. This is due to the fact that the 1000 people were sampled at random (so we shouldn’t expect our sample to differ from the entire population in a systematic way) and because the standard deviation of heights is not very large (if there were tremendous outliers in the data, such as 500 foot tall giants, we would need more samples to get an accurate estimate). This idea is made precise by the central limit theorem. It tells us how likely the true entire population mean is to fall different distances from our sample estimate, and says that the error of our estimate decreases like one over the square root of the size of our sample.

The same technique could work to approximate the mean weight of adult Americans, or age, or shoe size, or number of children. And in each case, the estimate would, with high probability, be quite accurate. We could even, if we liked, estimate all these quantities simultaneously if we collected all of this information about each of our 1000 people. But the more quantities we estimate, the greater the chance that at least one estimate is quite inaccurate. Since each estimate has some chance of being bad, if we make a sufficiently large number of estimates we should expect to get unlucky at some point and end up with one or more bad ones. So, if we aren’t just estimating mean height, but rather the mean of 50 different traits, we cannot claim that all 50 of these estimate are likely to be good. We should expect that some of them will be inaccurate, though we don’t know which ones.

This is where problems arise. Suppose that you are a researcher who is trying to find interesting differences between, say, southerners and northerners in the United States. Your dataset of 1000 adults contains 500 people from each group. What do you do? Well, it might seem reasonable to go ahead and compute the mean value of many different traits, and look at how these means differ between the two groups, to see if you can find any large differences that seem interesting. For instance, you may compute the average salary of each group, and see if they deviate from each other by a large enough amount to be deemed statistically significant. If they don’t, you can try another trait like IQ, or number of children, and repeat the process. If you try enough different traits, hopefully you’ll eventually find an intriguingly large difference between the groups.

The trouble is, we know that if you estimate a large number of quantities, some of them will be inaccurate, and so some of the apparent differences between your two groups may just be due to these inaccuracies. If you test enough traits, you will eventually find differences between the populations that look significant, even though it is just the result of chance.

In fact, even if northerners and southerners had no systematic differences between them, there would still be apparent differences that arose just from the particular sample of 1000 people you happened to have data on. For example, in your dataset, it just might happen that the northerners have lower numbers of children than southerners, even if this isn’t true for the underlying populations of all northerners and southerners. If you were to publish this finding, without making mention of the number of hypotheses you tested before finding it, it may seem that you had produced a meaningful result. In fact, the assessment of this result should take into account the number of hypotheses (e.g. northerners have smaller shoe sizes than southerners, northerners have greater salaries than southerners, etc.) that you tested before you discovered this one (and the p-values can be modified to include this information). The most significant seeming deviation between the groups found after testing 100 different hypotheses is very likely greatly inflated by chance. Whereas if you had only tested a small number of hypotheses against your data, and found a strong result, this would likely be a meaningful finding.

As a general rule, the greater the number of data points you have, the larger the number of quantities you can accurately estimate from your dataset. On a set of just 10 points, you may not even be able to get an accurate estimate of the mean value of a single trait (unless the trait had very slow standard deviation). Whereas on a dataset of a billion points, you probably could estimate dozens of quantities accurately.

Unfortunately, when you’re reading a paper, there is no way to tell how many hypotheses the researcher tested on his dataset unless he chooses to publish it. And there is a strong incentive to obscure this information. If a researcher releases the fact that he tested 20 hypotheses before finding 1 which was statistically significant, readers may discredit the result, or reviewers may reject it for publication. And if the researcher spent a lot of time and money collecting his dataset, it would feel like a waste to give up on the data just because his first five hypotheses tested on it don’t pan out. It might take a lot of restraint to not just keep testing hypothesis after hypothesis until he finds something publishable.

But even if researchers were excessively careful, that wouldn’t fully resolve the problem. When a hypothesis is confirmed by a dataset, we must consider whether it is truly a confirmation of the hypothesis being tested, or a result of the fact that 20 researchers tested 20 false hypotheses, and this one of the 20 happened to seem true by chance. That is, if enough hypotheses are tested over all, we may find a large number of false hypotheses among them that just happen to seem true.

What makes this problem more pernicious is that when a hypothesis fails to pan out, the result is often not published. This is due to the fact that hypothesis disconfirmations (e.g. “no association was found between cabbage eating and longevity”) are generally less interesting and harder to publish than confirmations (e.g. “an association was found between cabbage eating and longevity”). But since most new hypotheses in science turn out to be false, we should expect the number of negative results to be very large (except in situations where previously well validated results are being confirmed). Hence, the number of published test results will be much less than the number of total tests conducted, with test failures substantially underreported. So there is no good way to tell how many times a hypotheses failed to be confirmed by tests before one researcher finally ran one that seemed to confirm it. And if a very large number of false hypotheses are tested, but mostly just the ones that turn out to look true are published, you could end up with a field’s journals being flooded with false but seemingly verified hypotheses. In exploratory fields where almost all hypotheses are false, and where disconfirmations of a hypothesis are almost never published, you might even get into a situation where most published research findings are false.