To find out, at Clearer Thinking, we ran a massive study.

We tested thousands of people performing random subsets of 62 diverse cognitive tasks (vocab, math, logic, pattern recognition, reaction time, games, memorization, mental rotation, language learning, etc.)

We successfully replicated a classic finding: performance on nearly all cognitive tasks correlates positively with performance on the other tasks—a phenomenon known as the “positive manifold,” foundational to IQ.

IQ scores explained ~45% of variation across our diverse cognitive tasks, aligning with previous research. That’s very substantial for a single number (IQ score), but also far from capturing everything.

Some of the remaining 55% variance is pure noise; the rest likely comes from task skill (developed through practice) and task-specific aptitudes (likely influenced by both genetics and childhood experiences).

It’s also worth noting that while IQ is predictive of a diverse range of intelligence tasks, it doesn’t necessarily capture ALL that’s meant by intelligence. It’s unclear if it includes “street smarts,” social skills, or deep nature skills (like hunter-gatherers have).

We confirmed IQ predicts interesting outcomes:

• Actively open-minded thinking (r=0.43)

• Household income (weakly, r=0.15)

• Self-reported job performance (for lower IQ ranges only, r=0.48)

• Celebrity worship (substantially negatively correlated with IQ, r=-0.42)

• No link to happiness or life satisfaction.

IQ captures something—but WHAT? There’s no consensus. Theories include that IQ is…

• A single cognitive resource

• Working memory + executive control

• A measure of brain integration

• The result of overlapping cognitive processes

• The top of a hierarchy of abilities

Guess what’s a BETTER predictor of major life outcomes than IQ?

The Big Five personality traits.

In our data, openness, conscientiousness, extraversion, agreeableness, and neuroticism (when all 5 are used together) outperformed IQ at predicting most outcomes.

IQ clearly matters—it unavoidably jumps out from the data when testing people on a diverse range of cognitive tasks.

And yet…

IQ is far from being all that matters and is definitely not destiny.

We used the data from our study to create a new cognitive assessment, which analyzes ability across 7 dimensions and aims to provide you with useful insights about your mind.

If you’d like to take it, you can do so here:

https://programs.clearerthinking.org/cognitive-test-intro.html

(Proceeds support our mission.)

This piece was first written on March 25, 2025, and first appeared on my website on May 15, 2025.

On average, there is almost no correlation between human variables (see image). The average is only r=0.02, with the median even smaller at r=0.01. But this is misleading because it allows positive correlations to be canceled out by negative ones.

So let’s instead look at the distribution of the absolute values of the correlations. The average *magnitude* is r=0.1, with a median of 0.07. This shows how small a correlation of 0.1 is – it’s about the magnitude you’d get correlating two human variables picked at random!

We see similar results if we look only at positive correlations. The mean and median are still r=0.1 and r=0.07. So it seems that there *is* a “crud factor” – random human things really are correlated (on average)! But not very much, just slightly. 

Yet, while there’s some correlation on average, it’s also worth noting that the most *common* correlation is right around 0! So, the correlation mean (r=0.1), median (r=0.07), and mode (r≈0) are all a bit different.

Finally, note that different data sets could find different results depending on what they contain. But, to my knowledge, this is the most diverse and largest test of its kind by far.

But wouldn’t we measure a non-zero correlation even if there were no true correlations, just due to random chance? 

The answer is yes, but if there were no true correlations, the average absolute value of correlation would be a lot smaller. I ran a simulation where I generated fake correlations (where the true correlation was zero, any measured correlation was just due to noise arising from a limited sample size). I did this using the real sample sizes from our actual database of correlations. 

The result is that we still do measure an absolute value of correlation above zero, as expected, but it’s very small. The median absolute value of correlation is just 0.02 and the mean is r=0.03. We get pretty much the same numbers if we take the median and mean of just the positive correlations as well (instead of the absolute values). So even though a small amount of the crud factor likely is attributable to random noise, we can see that it is substantially bigger than what we’d expect if it was all just due to random noise.

You can explore our 1 million correlations for free with our Personality Map tool: https://personalitymap.io

To learn more about the “crud factor” in psychology, check out this interesting paper:
Crud (Re)Defined.

The reason stems from my thinking that all of the following are important and true (and too often overlooked) regarding science:

1) A lot of science is real AND valuable to society.

2) A lot of “science” is actually fake – see, for instance, a decent percentage of papers in psychology 15 years ago.

3) “Science” (as an approach to knowledge discovery) is one of humanity’s greatest inventions – but in practice, it is reasonably often misapplied, or the process is distorted due to bad incentives or poor training. Unfortunately, not all fields of science have done a good job of being self-correcting either, so sometimes, fields go in bad directions for quite a while and need reform. There are different kinds of bad science:

(i) Sometimes, science is “bad” because it uses unsound methods for figuring out the truth (such as when p-hacking is rampant).

(ii) Sometimes it is “bad” because it overclaims (e.g., “Importance Hacking” where scientists claim they found something important/valuable when they didn’t actually demonstrate what they claim in their study. Or cases where science is used to “prove” questions that can’t be proven by science – such as which policy is better in a particular context when it’s actually a tradeoff between different values).

(iii) Other times science is bad because it is biased (e.g., when people are only willing to run or publish studies that show X but not that show the opposite of X).

(iv) And sometimes science is bad because it’s simply fraudulent.

4) Promoting broad “trust the science” is misguided (and actually harmful) because a bunch of science is fake. If you tell people to always just “trust the science,” then you are going to cause them to be tricked by a bunch of bad science, or you are going to contribute to their disillusionment and loss of trust when they discover (correctly) that some of the science you’re saying is good is actually garbage.

5) The “distrust all science” view is probably an even worse take than “trust the science.” If you distrust all science, you are likely to miss out on incredible things (such as highly effective treatments), and you set yourself up to fall for tons of things that don’t work (e.g., widely used unscientific treatments). Those who tell people to always just “trust the science” sometimes accidentally push people into the “distrust all science” view when those people realize that some of what they are being told to trust is crap.

6) So, hard as it is, rather than promoting either “trust all science” or “distrust all science,” the course of action I believe in with regard to science education is to teach people that “Science” (as a method) is an incredibly powerful and useful invention, but that “science” (as actually practiced) is much like every other field: some of it is good, some of it is crap. There are good hairdressers and bad hairdressers, and there is good science and bad science (and unfortunately, some bad science ends up in the very top journals – while journals and peer review absolutely do block some bad science, they unfortunately still let through quite a lot of it).

Since some science is well done, and some of it is poorly done, it’s very valuable to learn to tell the difference to make the best use of scientific results – both with regard to applying it in your own life and using it to form your beliefs about the world.

If we pretend science is all good or all bad, we do a lot of harm. We need nuance to see through the bad stuff while maintaining the tremendous benefits.

This piece was first written on November 20, 2024, and first appeared on my website on January 14, 2025.

You have probably heard the phrase “replication crisis.” It refers to the grim fact that, in a number of fields of science, when researchers attempt to replicate previously published studies, they fairly often don’t get the same results. The magnitude of the problem depends on the field, but in psychology, it seems that something like 40% of studies in top journals don’t replicate. We’ve been tackling this crisis with our new Transparent Replications project, and this post explains one of our key ideas.

Replication failures are sometimes simply due to bad luck, but more often, they are caused by p-hacking – the use of fishy statistical techniques that lead to statistically significant (but misleading or erroneous) results. As big a problem as p-hacking is, there is another substantial problem in science that gets talked about much less. Although certain subtypes of this problem have been named previously, to my knowledge, the problem itself has no name, so I’m giving it one: “Importance Hacking.”

Academics want to publish in the top journals in their field. To understand Importance Hacking, let’s consider a (slightly oversimplified) list of the three most commonly-discussed ways to get a paper published in top psychology journals:

1. Conduct valuable research – make a genuinely interesting or important discovery, or add something valuable to the state of scientific knowledge. This is, of course, what just about everyone wants to do, but it’s very, very hard!
2. Commit fraud – for instance, by making up your data. Thankfully, very few people are willing to do this because it’s so unethical. So this is by far the least used approach.
3. p-hack – use fishy statistics, HARKing (i.e., hypothesizing after the results are known), selective reporting, using hidden researcher degrees of freedom, etc., in order to get a p<0.05 result that is actually just a false positive. This is a major problem and the focus of the replication crisis. Of course, false positives can also come about without fault, due to bad luck.

But here is a fourth way to get a paper published in a top journal: Importance Hacking.

4. Importance Hack – get a result that is actually not interesting, not important, and not valuable, but write about it in such a way that reviewers are convinced it is interesting, important, and/or valuable, so that it gets published.

For research to be valuable to society (and, in an ideal world, publishable in top journals), it must be true AND interesting (or important, useful, etc.). Researchers sometimes p-hack their results to skirt around the “true” criterion (by generating interesting false positives). On the other hand, Importance Hacking is a method for skirting the “interesting” criterion.

Importance Hacking is related to concepts like hype and overselling, though hype and overselling are far more general. Importance Hacking refers specifically to a phenomenon whereby research with little to no value gets published in top journals due to the use of strategies that lead reviewers to misinterpret the work. On the other hand, hype and overselling are used in many ways in many stages of research (including to make valuable research appear even more valuable).

One way to understand importance hacking is by comparing it to p-hacking. P-hacking refers to a set of bad research practices that enable researchers to publish non-existent effects. In other words, p-hacking misleads paper reviewers into thinking that non-existent effects are real. Importance Hacking, on the other hand, encompasses a different set of bad research practices: those that lead paper reviewers to believe that real (i.e., existent) results that have little to no value actually have substantial value.

This diagram illustrates how I think Importance Hacking interferes with the pipeline of producing valuable research:

There are a number of subtypes of Importance Hacking based on the method used to make a result appear interesting/important/valuable when it’s not. Here is how I subdivide them:

Types of Importance Hacking

1. Hacking Conclusions: make it seem like you showed some interesting thing X but actually show something else (X′) which sounds similar to X but is much less interesting/important. In these cases, researchers do not truly find what they imply they have found. This phenomenon is also closely connected with validity issues.

• Example 1: showing X is true in a simple video game but claiming that X is true in real life.
• Example 2: showing A and B are correlated and claiming that A causes B (when really A and B are probably both caused by some third factor C, which makes the finding much less interesting).
• Example 3: if a researcher claims to be measuring “aggression,” and couches all conclusions in these terms but is actually measuring milliliters of hot sauce that a person puts in someone else’s food. Their result about aggression will be valid only insofar as it is true that this is a valid measure of aggression.
• Example 4: some types of hacking conclusions would fall under the terms “overclaiming” or “overgeneralizing;” Tal Yarkoni has a relevant paper called The Generalizability Crisis.

2. Hacking Novelty: refer to something in a way that makes it seem more novel or unintuitive than it is. Perhaps the result is already well known or is merely what just about everyone’s common sense would already tell them is true. In these cases, researchers really do find what they claim to have found, but what they found is not novel (despite them making it seem so). Hacking Novelty is also connected to the “Jingle-jangle” fallacy – where people can be led to believe two identical concepts are different because they have different names (or, more subtly, because they are operationalized somewhat differently).

• Example 1: showing something that is already well-known but giving it a new name that leads people to think it is something new. The concept of “grit” has received this criticism; some people claim it could turn out to be just another word for conscientiousness (or already known facets of conscientiousness) – though this question does not yet seem to be settled (different sides of this debate can be found in these papers: 1, 2, 3 and 4).
• Example 2: showing that A and B are correlated, which seems surprising given how the constructs are named, but if you were to dig into how A and B were measured, it would be obvious they would be correlated.
• Example 3: showing a common-sense result that almost everyone already would predict but making it seem like it’s not obvious (e.g., by giving it a fancy scientific name).

3. Hacking Usefulness: make a result seem useful or relevant to some important outcome when in fact, it’s useless and irrelevant. In these cases, researchers find what they claim to have found, but what they find is not useful (despite them making it sound useful).

• Example: focusing on statistical significance when the effect size is so small that the result is useless. Clinicians often distinguish between “statistical significance” and “clinical significance” to highlight the pitfalls of ignoring effect sizes when considering the importance of a finding.

4. Hacking Beauty: make a result seem clean and beautiful when in fact, it’s messy or hard to interpret. In these cases, researchers focus on certain details or results and tell a story around those, but they could have focused on other details or results that would have made the story less pretty, less clear-cut, or harder to make sense of. This is related to Giner-Sorolla’s 2012 paper Science or art: How aesthetic standards grease the way through the publication bottleneck but undermine science. Hacking beauty sometimes reduces to selective reporting of some kind (i.e., selective reporting of measures, analyses, or studies) or at least of selective focus on certain findings and not others. This becomes more difficult with pre-registration; if you have to report the results of planned analyses, there’s less room to make them look pretty (you could just say they’re pretty, but that seems like overclaiming)

• Example: emphasizing the parts of the result that tell a clean story while not including (or burying somewhere in the paper) the parts that contradict that story

Science faces multiple challenges. Over the past decade, the replication crisis and subsequent open science movement have greatly increased awareness of p-hacking as a problem. Measures have begun to be put in place to reduce p-hacking. Importance Hacking is another substantial problem, but it has received far less attention.

If a pipe is leaking from two holes and its pressure is kept fixed, then repairing one hole will result in the other one leaking faster. Similarly, as best practices increasingly become commonplace as a means to reduce p-hacking, so long as the career pressures to publish in top journals don’t let up, the occurrence of Importance Hacking may increase.

It’s time to start the conversation about how Importance Hacking can be addressed.

If you’re interested in learning more about Importance Hacking, you can listen to psychology professor Alexa Tullett and me discussing it on the Clearer Thinking podcast (there, I refer to it as “Importance Laundering,” but I now think “Importance Hacking” is a better name) or me talking about it on the Two Psychologists Four Beers podcast. We also discuss my new project, Transparent Replications, which conducts rapid replications of recently published psychology papers in top journals in an effort to shift incentives and create more reliable, replicable research. If you enjoyed this article, you may be interested in checking our replication reports and learning more about the project.

Did you like this article? If so, you may like to explore the ClearerThinking Podcast, where I have fun, in-depth conversations with brilliant people about ideas that matter. Click here to see a full list of episodes.

The oddest part is that despite a heroic attempt to do so, the study authors failed to explain why the different research teams reached such different conclusions:

“More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation.”

The hypothesis they were trying to test was whether greater immigration reduces support for social policies (such as for government-provided healthcare).

The study included data from 31 countries at up to five time periods each.

If all of the countries and time points were independent data points, that would be an effective sample size of at most 5*31 = 155.

However, the data points from one country at different points in time are highly correlated.

So, in practice, this might be equivalent to more like 75 (independent) data points.

If the (effective) sample size was equivalent to only about n=75, the 95th percentile confidence interval on a correlation could be pretty large (e.g., it could be +- 0.20), suggesting that false negatives would be very common (unless the relationship in question is pretty strong).

In view of this, I think it’s possible that this study was doomed from the start.

Why? Unless we’d expect a reasonably strong effect, maybe there just wasn’t enough data to answer the question at hand.

Some of the teams may have come to this conclusion as well. Of the 73 research teams involved in the study, one of them conducted preliminary measurement scaling tests and concluded that the hypothesis could not be reliably tested. Another team’s preregistered models failed to converge, and so that team also had no numerical results. Some teams reached more than one conclusion, and across the 89 team conclusions reached, 12 of those conclusions (13.5%) were that the hypothesis was not testable with the data provided.

Then, of course, there is the possibility of confounding variables – other factors that could be linked to both immigration and inflation that make the true relationship seem bigger or smaller than it really is.

Here’s a link to the paper if you’re interested: “Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty.”

A big thanks to Cameron Colby Thomson for pointing me to this study!

This piece was first written on November 30, 2022, and first appeared on this site on September 29, 2023.

• If you’re a psychology researcher investigating possible causes of depression (Y), you may have trouble disentangling the effects of poor sleep quality (A) and anxiety (B), both of which tend to be correlated with depression.
• If you’re a health researcher investigating the causes of diabetes (Y), you may have trouble disentangling the effects of high carbohydrate intake (A) and obesity (B).
• If you’re investigating the causes of high life satisfaction (Y), you may have trouble disentangling the effects of friendship quality (A) and mental well-being (B).

In all these examples, we know that at least two of the variables (A and B) are related to the main variable (Y), but the really tricky question is to figure out what all the possible causal relationships are between the three. For instance, does A cause B, which causes Y, does Y cause both A and B, or is there some other explanation? 

In the pdf below, I sketch out 45 possible explanations to consider in situations where there are two variables that both correlate with a third variable of interest.

First of all, there are the types of causal relationships one often expects, where A and B both cause Y in simple ways (either directly or through each other):

Even if A and B really do cause Y, they could be interconnected to each other in complex ways:

It also could be the case that only A or only B causes Y, with the other variable only appearing to cause Y due to a confounding effect:

It’s also possible that Y is actually one of the causes rather than merely being caused by A and B:

Then there are situations where there is a critical other variable (or set of variables – represented as a “?” below) that are integral to the causal structure:

Finally, there are situations where Y is caused by A or B (or both), but Y also causes A or B (or both), resulting in a cyclic relationship:

Here’s a link to my pdf showing most of the possible relationships.

If you want to read about other challenges associated with untangling causality in the real world, you can read another post about this here.

I first created this diagram on April 19, 2021. I made minor edits to the diagram and wrote this piece with assistance from Clare Harris. This piece first appeared on my website on November 22, 2023.