To find out, at Clearer Thinking, we ran a massive study.

We tested thousands of people performing random subsets of 62 diverse cognitive tasks (vocab, math, logic, pattern recognition, reaction time, games, memorization, mental rotation, language learning, etc.)

We successfully replicated a classic finding: performance on nearly all cognitive tasks correlates positively with performance on the other tasks—a phenomenon known as the “positive manifold,” foundational to IQ.

IQ scores explained ~45% of variation across our diverse cognitive tasks, aligning with previous research. That’s very substantial for a single number (IQ score), but also far from capturing everything.

Some of the remaining 55% variance is pure noise; the rest likely comes from task skill (developed through practice) and task-specific aptitudes (likely influenced by both genetics and childhood experiences).

It’s also worth noting that while IQ is predictive of a diverse range of intelligence tasks, it doesn’t necessarily capture ALL that’s meant by intelligence. It’s unclear if it includes “street smarts,” social skills, or deep nature skills (like hunter-gatherers have).

We confirmed IQ predicts interesting outcomes:

• Actively open-minded thinking (r=0.43)

• Household income (weakly, r=0.15)

• Self-reported job performance (for lower IQ ranges only, r=0.48)

• Celebrity worship (substantially negatively correlated with IQ, r=-0.42)

• No link to happiness or life satisfaction.

IQ captures something—but WHAT? There’s no consensus. Theories include that IQ is…

• A single cognitive resource

• Working memory + executive control

• A measure of brain integration

• The result of overlapping cognitive processes

• The top of a hierarchy of abilities

Guess what’s a BETTER predictor of major life outcomes than IQ?

The Big Five personality traits.

In our data, openness, conscientiousness, extraversion, agreeableness, and neuroticism (when all 5 are used together) outperformed IQ at predicting most outcomes.

IQ clearly matters—it unavoidably jumps out from the data when testing people on a diverse range of cognitive tasks.

And yet…

IQ is far from being all that matters and is definitely not destiny.

We used the data from our study to create a new cognitive assessment, which analyzes ability across 7 dimensions and aims to provide you with useful insights about your mind.

If you’d like to take it, you can do so here:

https://programs.clearerthinking.org/cognitive-test-intro.html

(Proceeds support our mission.)

This piece was first written on March 25, 2025, and first appeared on my website on May 15, 2025.

How to Test a Theory Without Doing An Experiment:

Think of as many things as you can that are predicted by your theory, and try to identify at least one prediction that has the following four properties:

(1) your theory implies it is very likely to be true.

(2) it seems very unlikely to be true if your theory is false (i.e., there is no reason you can see to make that prediction other than if you believe in your theory). You may want to ask others if they agree that the prediction seems very unlikely to be true from the perspective of those who don’t believe your theory; otherwise, you might be deceiving yourself.

(3) you don’t already know the answer to whether it is actually true or not (you are merely guessing it is true because it is implied by your theory). This is important because if you already know the answer, you may be tricking yourself one way or another (e.g., by having already used this evidence in the construction of your theory, or by choosing this particular test because on some level you know it will make your theory look good).

(4) you can reliably look up whether that prediction is true or not. Then all you have to do is look up whether the prediction is true!

If it turns out to be true, that provides evidence to support your theory. More specifically, the more likely that true result is given your theory, and the less likely that true result is (to someone who doesn’t know your theory or if your theory is invalid), the more evidence the test provides. To explore the math for only a moment: that test of your theory should change your prior odds (i.e., your estimated probability that your theory is true, divided by the probability that it is not true) by using what’s known as the “Bayes factor,” namely, your estimation probability. Looking up the answer to that question would yield the result it did, if your theory is true divided by the probability it would yield that result if your theory is not true.

In other words:

odds your theory is true = ( probability of that result if your theory is true ÷ probability of that result if your theory is false) × prior odds your theory is true from before the test

Odds can be confusing to think about (remember that it is the probability that something is true divided by the probability that it is false). For instance, odds of 2 would mean that a thing is twice as likely to be the case then it is to not be the case.

It’s worth noting that the above procedure can work really effectively to test your hypotheses for yourself, but it’s NOT a good way to provide evidence that your theory is true to other people that don’t trust you (i.e., your honest pursuit of truth or your competence). The reason is that it’s easy to claim you conducted this fair test of your theory, when in fact, you already knew the answer to how the test would turn out (and therefore were incentivized to choose this particular test of your theory rather than another). Or perhaps you interpreted your theory in a weird way or tweaked it after the fact to make it look like it correctly predicted the outcome when it actually didn’t. Or maybe you screwed up one of the steps accidentally. In other words, the evidence this procedure produces is only trustworthy if you really trust the rigor and character of the person who produced the evidence.

This is why experiments are, at least in theory (but unfortunately too often not in practice), a better way to gather evidence when you need to communicate those results to others who don’t trust you, especially if you pre-register your study and analysis plan, and if you release your materials and the data, you collect.

If you’re interested in the concepts underpinning the approach I’ve described here, you may want to check out our program on Bayesian thinking (which you may or may not be happy to know uses almost no math):

http://programs.clearerthinking.org/question_of_evidence.ht…

After noticing a number of times that people’s feelings about formal credentials can differ dramatically and that this seems to impact their views on certain important topics, I’ve been working on defining a “credentialist” trait (or attitude).

In a nutshell, the non-credentialist/credentialist spectrum, as I’m defining it, captures how important a person thinks formal credentials are, as well as how they feel those credentials should influence who we should trust and who should express opinions (e.g., should only formalized experts comment on a topic, or is it good for non-experts to comment as well?)

I developed a 4-minute test to measure the trait, so if you’d like to find out if you are a “credentialist” or “non-credentialist” (which I define as being in the top or bottom 20th percentile of each trait) or find out your own credentialist score, you can take the test here:

https://programs.clearerthinking.org/credentialist_test.html

Here are simplified/extreme prototypes to illustrate the distinction (few people are as extreme as these prototypes):

*Credentialists* get annoyed when someone without the right credential is giving their opinion on a topic, are impressed by formal degrees (e.g. PhDs and MDs), do not like it when non-experts have their own personal theory about a topic, trust people a lot more when they have formal credentials, think its unlikely someone could get really good at a complex topic without formal training, think that non-experts should not contradict experts, would go to school to learn a topic if they wanted to get good at it, find it annoying if startup founders talk about disrupting industries they have not already worked in, tend to describe people in terms of their schooling and job history (rather than, for example, their personality), and trust the opinion’s of people a lot more if they went to an excellent college.

*Non-credentialists* think that it fine (or even good) to express opinions when you’re a non-expert, are not particularly impressed by formal degrees, don’t view degrees or certifications as a strong indicator of trust, think it’s fine (or even good) for non-experts to criticize the views of experts, are prone to teach themselves material rather than going to school for it, don’t mind startup founders attempting to disrupt industries from the outside, tend not to describe people in terms of their schooling and job history, and don’t view the quality of the college a person went to as a significant factor in whether to trust their opinions.

I measured the trait on a 0 to 1 scale, and on the 143 people I collected it for, it has a mean of 0.55 and a standard deviation of 0.16, with a pretty nice bell curve shape.

Interestingly, I found a very low correlation between credentialist scores and education [r=0.03], identifying as female [r=-0.04], and income [r=0.03], and little correlation with age [r=-.08]. This suggests that credentialist differences have little to do with demographic characteristics!

Furthermore, whether you are a credentialist does not even seem to have that much to do with whether you yourself have credentials, as responses to the question “I myself have substantial formal credentials” had a correlation of only about r=0.13 with the credentialist scores. It also has little to do with how ambitious people are, as responses to “I have highly ambitious goals for what I will achieve in my life” had a correlation of only r=0.07 to credentialist scores.

Of the 22 questions I tested people’s agreement on in order to measure the trait, the 2 most effective questions (in the sense that they correlate highly with the average of the other questions but don’t correlate very highly with each other) are:

Q1: It annoys me when someone without the right credential is giving their opinion on a topic (e.g., a non-doctor commenting about medicine or a non-accountant commenting on accounting)

[r=0.71 against the average of the other 21 questions]

and

Q2: I am very impressed by formal degrees (e.g., PhDs, MDs, JDs, etc.)

[r=0.54 against the average of the other 21 questions, yet a fairly low r=0.24 with respect to Q1]

The 22 questions I developed hang together nicely and point in generally the same direction. Basic factor analysis revealed only one main factor in the questions, and the question least correlated to the others still had a positive correlation of r=0.34 with the average of the other questions (which was “If I heard that someone had won a prize in their field, I would think very highly of it”).

Most people fall in the middle of this trait, of course (e.g., viewing credentials as at least somewhat positive but the lack of them not that negative), without an extreme viewpoint either way. However, here are some anonymized qualitative responses I collected from people at the tail ends of the spectrum:

Credentialists:

“I don’t respect people who don’t have formal credentials, and one of my biggest pet peeves is people–even really intelligent people–speaking about things they aren’t experts on. Just because someone is known, or even renowned in another field, doesn’t place their opinion on another topic anywhere above another non-educated person.”

“People with formal credentials have generally gone through a rigorous peer review process that demands a considerable depth of understanding and knowledge.”

“People who don’t have credentials have no business talking about things they know nothing about. That’s how misinformation gets spread, and misinformation is harmful to society as a whole.”

Non-credentialists:

“I think that people can have a valid voice no matter their level of formal schooling. The opposite also holds true: a degree is not necessarily representative of ability.”

“I feel that colleges have become scams. They are money sponges that make you pay for your own brainwashing. I value intelligence and how well-read someone is on a subject, and we don’t need a self-appointed team of left-wing experts to ‘allow’ us to do that for ourselves anymore. People with credentials have more money than sense and have been taught what NOT to think more than they have been taught HOW to think.”

“While I respect those who have worked to earn professional credentials, people who are self-taught oftentimes know much more about a subject than someone with an expensive degree.”

Note: designing this scale and writing this post is a very non-credentialist thing to do since I’m not a social scientist.

This piece was first written on July 21, 2017, and first appeared on my website on June 10, 2025.

Let’s take an example to illustrate the principle. Suppose that you have information about 1000 people selected at random from the U.S. adult population. Your dataset includes these people’s heights, weights, ages, shoe sizes, and so forth. Now, if your goal is to know the mean height of all people in America, you can produce an estimate of this quantity by averaging the heights of the 1000 people you have information about. Despite the fact that your sample contains just 1000 people, rather than the full set of 230,000,000 or so American adults of interest, your estimate will, with high probability, be within a couple of inches of the total population mean height. This is due to the fact that the 1000 people were sampled at random (so we shouldn’t expect our sample to differ from the entire population in a systematic way) and because the standard deviation of heights is not very large (if there were tremendous outliers in the data, such as 500 foot tall giants, we would need more samples to get an accurate estimate). This idea is made precise by the central limit theorem. It tells us how likely the true entire population mean is to fall different distances from our sample estimate, and says that the error of our estimate decreases like one over the square root of the size of our sample.

The same technique could work to approximate the mean weight of adult Americans, or age, or shoe size, or number of children. And in each case, the estimate would, with high probability, be quite accurate. We could even, if we liked, estimate all these quantities simultaneously if we collected all of this information about each of our 1000 people. But the more quantities we estimate, the greater the chance that at least one estimate is quite inaccurate. Since each estimate has some chance of being bad, if we make a sufficiently large number of estimates we should expect to get unlucky at some point and end up with one or more bad ones. So, if we aren’t just estimating mean height, but rather the mean of 50 different traits, we cannot claim that all 50 of these estimate are likely to be good. We should expect that some of them will be inaccurate, though we don’t know which ones.

This is where problems arise. Suppose that you are a researcher who is trying to find interesting differences between, say, southerners and northerners in the United States. Your dataset of 1000 adults contains 500 people from each group. What do you do? Well, it might seem reasonable to go ahead and compute the mean value of many different traits, and look at how these means differ between the two groups, to see if you can find any large differences that seem interesting. For instance, you may compute the average salary of each group, and see if they deviate from each other by a large enough amount to be deemed statistically significant. If they don’t, you can try another trait like IQ, or number of children, and repeat the process. If you try enough different traits, hopefully you’ll eventually find an intriguingly large difference between the groups.

The trouble is, we know that if you estimate a large number of quantities, some of them will be inaccurate, and so some of the apparent differences between your two groups may just be due to these inaccuracies. If you test enough traits, you will eventually find differences between the populations that look significant, even though it is just the result of chance.

In fact, even if northerners and southerners had no systematic differences between them, there would still be apparent differences that arose just from the particular sample of 1000 people you happened to have data on. For example, in your dataset, it just might happen that the northerners have lower numbers of children than southerners, even if this isn’t true for the underlying populations of all northerners and southerners. If you were to publish this finding, without making mention of the number of hypotheses you tested before finding it, it may seem that you had produced a meaningful result. In fact, the assessment of this result should take into account the number of hypotheses (e.g. northerners have smaller shoe sizes than southerners, northerners have greater salaries than southerners, etc.) that you tested before you discovered this one (and the p-values can be modified to include this information). The most significant seeming deviation between the groups found after testing 100 different hypotheses is very likely greatly inflated by chance. Whereas if you had only tested a small number of hypotheses against your data, and found a strong result, this would likely be a meaningful finding.

As a general rule, the greater the number of data points you have, the larger the number of quantities you can accurately estimate from your dataset. On a set of just 10 points, you may not even be able to get an accurate estimate of the mean value of a single trait (unless the trait had very slow standard deviation). Whereas on a dataset of a billion points, you probably could estimate dozens of quantities accurately.

Unfortunately, when you’re reading a paper, there is no way to tell how many hypotheses the researcher tested on his dataset unless he chooses to publish it. And there is a strong incentive to obscure this information. If a researcher releases the fact that he tested 20 hypotheses before finding 1 which was statistically significant, readers may discredit the result, or reviewers may reject it for publication. And if the researcher spent a lot of time and money collecting his dataset, it would feel like a waste to give up on the data just because his first five hypotheses tested on it don’t pan out. It might take a lot of restraint to not just keep testing hypothesis after hypothesis until he finds something publishable.

But even if researchers were excessively careful, that wouldn’t fully resolve the problem. When a hypothesis is confirmed by a dataset, we must consider whether it is truly a confirmation of the hypothesis being tested, or a result of the fact that 20 researchers tested 20 false hypotheses, and this one of the 20 happened to seem true by chance. That is, if enough hypotheses are tested over all, we may find a large number of false hypotheses among them that just happen to seem true.

What makes this problem more pernicious is that when a hypothesis fails to pan out, the result is often not published. This is due to the fact that hypothesis disconfirmations (e.g. “no association was found between cabbage eating and longevity”) are generally less interesting and harder to publish than confirmations (e.g. “an association was found between cabbage eating and longevity”). But since most new hypotheses in science turn out to be false, we should expect the number of negative results to be very large (except in situations where previously well validated results are being confirmed). Hence, the number of published test results will be much less than the number of total tests conducted, with test failures substantially underreported. So there is no good way to tell how many times a hypotheses failed to be confirmed by tests before one researcher finally ran one that seemed to confirm it. And if a very large number of false hypotheses are tested, but mostly just the ones that turn out to look true are published, you could end up with a field’s journals being flooded with false but seemingly verified hypotheses. In exploratory fields where almost all hypotheses are false, and where disconfirmations of a hypothesis are almost never published, you might even get into a situation where most published research findings are false.