But part of what makes me an extreme non-credentialist is that if I spend an hour watching someone with a nutrition Ph.D. debate a completely self-taught person, and the Ph.D. is making bad arguments and pointing to weak evidence, and the self-taught person is making very solid arguments and pointing to strong evidence and has a very solid command of the relevant facts, the fact that the first person has a Ph.D. will be nearly completely washed out for me at that point, and I will trust the second person’s view of nutrition far more based on the quality of their thinking and the reasons underlying why they believe what they do.

So, being a non-credentialist to me isn’t about thinking that credentials are meaningless, but rather, it involves being willing to quickly update away from the evidence of a credential once you have more direct evidence about the way a person comes to conclusions and what they know.

Most Ph.D.s in a subject are vastly more reliable sources of information on that subject than most non-Ph.D.s on that same subject, but there are lots of exceptions, and sometimes self-taught people are absolutely world-class (and, in any human endeavor, plenty of people with fancy credentials are actually full of B.S.)

Another thing that makes me a non-credentialist is that I love to see highly credible, highly knowledgeable, self-taught people discussing topics and spreading their ideas (whereas some people are very much rubbed the wrong way when someone is talking publicly about a topic they lack a credential in).

An important note: when there is a strong scientific consensus, that is usually a strong starting point for beliefs on topics you know little about (e.g., in physics or biology), even though the consensus is not always right. But trusting the scientific consensus is not the same as trusting one person due to their credentials – a strong scientific consensus is typically more reliable than individual experts.

If you’d like to figure out how much of a credentialist or non-credentialist you are, you can take our credentialist test here.

This piece was first written on February 28, 2024, and first appeared on my website on March 22, 2024.

You can make inferences using…

(1) Deduction:

As a consequence of the definition of X and Y, if X then Y.

X applies to this case. Therefore Y.

“Plato is a man, and all men are mortal; therefore Plato is mortal.”

“For any number that is an integer, there exists another integer greater than that number. 1,000,000 is an integer. So there exists an integer greater than 1,000,000.”

Especially popular among philosophers and mathematicians?

Flaws: to apply to the world, you need to add in assumptions about the world, or to apply other methods of inference on top.

(2) Frequencies:

In the past, 95% of the time that X occurred, Y occurred.

X occurred. Therefore Y (with high probability).

“95% of the time when we saw a transaction identical to this one, it was fraudulent. So this transaction is fraudulent.”

Especially popular among applied statisticians and data scientists?

Flaws: You need to have a moderately large number of examples like the current one to perform calculations on, and the method assumes that those past examples were drawn from a process that is (statistically) just like the one that generated this latest example. Moreover, sometimes it is unclear what it means for “X” to have occurred. What if it’s something that’s very similar to but not quite like X that occurred – should that be counted? If we broaden our class of what counts or change to another class that still encompasses all of our prior examples, we’ll potentially get a different answer. Though, fortunately, there are plenty of cases where the class to use is fairly obvious.

(3) Models:

Given our probabilistic model of this thing, when X occurs, the probability of Y is 0.95.

X occurred. Therefore Y (with high probability).

“Given our multivariate Gaussian model of loan prices, when this loan defaults, there is a 0.95 probability of this other loan defaulting.”

Especially popular among financial engineers and risk modelers?

Flaws: hinges on the appropriateness of the parameterized probabilistic model chosen, may require a moderately large amount of past data to estimate free model parameters, and may go haywire if modeling assumptions are suddenly violated.

(4) Regression:

In prior data, as X and Z increased, the likelihood of Y increased.

X and Z are at high levels. Therefore Y.

“Height for children can be approximately predicted as an (increasing/positive) linear function of age and weight. This child is older and heavier than the others, so we predict he is also taller than the others.”

Especially common among economists and data scientists?

Flaws: often is applied with simple assumptions (e.g., linearity) that may not capture the complexity of the inference, but very large amounts of data may be needed to apply much more complex models (e.g., to use neural networks).

(5) Bayesianism:

Given my prior odds on Y being true…

And given evidence X…

And given my Bayes factor, which is my estimate of how much more likely X is to occur if Y is true than if Y is not true…

I calculate that Y is far more likely to be true than to not be true (by multiplying the prior odds by the Bayes factor to get the posterior odds).

Therefore Y (with high probability).

“My prior odds that my boss is angry at me were 1 to 4, because he’s angry at me about 20% of the time. But then he came into my office shouting and flipped over my desk, which I estimate is 200 times more likely to occur if he’s angry at me compared to if he’s not. So now the odds of him being angry at me are 200 * (1/4) = 50 to 1 in favor of him being angry.”

Not as popular as it should be?

Flaws: it is sometimes hard to know what to set your prior odds at, and it can be very hard in some cases to perform the calculation. In practice, carrying out the calculation might end up relying on subjective estimates of the odds, which can be especially tricky to guess when the evidence is not binary (i.e., not of the form “happened” vs. “didn’t happened”), or if you have lots of different pieces of evidence that are partially correlated. On the other hand, if you can do the calculations in a given instance, and have a sensible way to set a prior, this is, in my opinion, the mathematically optimal framework to use for probabilistic prediction. In that sense, we can think of many of the other approaches on this list as (hopefully pragmatic) approximations of Bayesianism (sometimes good approximations, sometimes bad ones).

(6) Theories:

Given our theory, when X occurs, Y occurs.

X occurred. Therefore Y.

“One theory is that depressed people are most at risk for suicide when they are beginning to come out of a really bad depressive episode. So as depression is remitting, patients should be carefully screened for potentially increasing risk factors.”

“When inflation rises, unemployment falls. Inflation is rising, so unemployment will fall.”

Especially popular among psychologists and economists?

Flaws: it’s very challenging to come up with reliable theories, and often you will not know how accurate such a theory is. Even if it has substantial truth to it and is often right, there may be cases where the opposite of what was predicted actually happens, and for reasons that the theory can’t explain.

(7) Causes:

We know that X causes Y to occur.

X occurred. Therefore Y.

“Rusting of gears causes increased friction, leading to greater wearing down. In this case, the gears were heavily rusted, so we expect to find a lot of wearing down.”

“This gene produces this phenotype, and we see that this gene is present, so we expect to see the phenotype.”

Especially popular among engineers and biologists?

Flaws: it’s often extremely hard to figure out causality in a highly complex system, especially in “softer” subjects like nutrition.

(8) Experts:

This expert (or prediction market, or prediction algorithm) X is 90% accurate at predicting things in this general domain of prediction.

X predicts Y. Therefore Y (with high probability).

“This prediction market has been right 90% of the time when predicting recent baseball outcomes, and in this case, they predict that the Yankees will win.”

Not as popular as it should be?

Flaws: you often don’t have access to the predictions of experts (or of prediction markets or prediction algorithms), and when you do, you usually don’t have reliable measures of their past accuracy.

(9) Metaphors:

X, which is what we are dealing with now, is metaphorically a Z.

For Z, when W is true, then obviously Y.

Now W (or its metaphorical equivalent) is true for X. Therefore Y.

“Your life is but a boat, and you are riding on the waves of your experiences. When a raging storm hits, a boat can’t be under full sail. It can’t continue at its maximum speed. You are experiencing a storm now, and so you too must learn to slow down.”

Especially popular among self-help gurus and some ancient philosophers?

Flaws: Z working as a metaphor for X doesn’t mean that all (or even most) solutions that are good for situations involving Z are appropriate (or even make any sense) for X.

(10) Similarities:

X occurred, and X is very similar to Z in properties A, B, and C.

When things similar to Z in properties A, B, and C occur, Y usually occurs.

Therefore Y (with high probability).

“This conflict is similar to the Gulf war in that…and with “Gulf”-like wars, we have always seen that…”

“This data point (with unknown label) is closest in feature space to this other data point which is labeled ‘cat,’ and all the other labeled points around that point are also labeled ‘cat,’ so this unlabeled point should also likely get the label ‘cat.’”

Especially popular among historians and within some machine learning algorithms?

Flaws: in the history case, it is difficult to know which features are the appropriate ones to use to compare similarities, and often the conclusions are based on a relatively small number of examples. In the machine learning case, a very large amount of data may be needed to train the model.

(11) Cases:

In this handful of examples (or perhaps even just one example) where X occurred, Y occurred.

X occurred. Therefore Y.

“The last time we elected a [insert political group you don’t like] as president, we saw what happened. Let’s not make that mistake again.”

“The last three times I went to action movies, I didn’t like them. So I don’t want to go to one again.”

Especially popular with politicians and with nearly everyone in daily living?

Flaws: unless we are in a situation with very little noise/variability, a few examples likely will not be enough to accurately generalize from.

(12) Intuition:

X occurred. My intuition (that I may have trouble explaining) predicts that when X occurs, Y is true. Therefore Y.

“The tone of voice he used when he talked about his family gave me a bad vibe. My feeling is that anyone who talks about their family with that tone of voice probably does not really love them.”

Popular with nearly everyone in daily living?

Flaws: our intuitions can be very well-honed in situations we’ve encountered many times and that we received feedback on (i.e., where there was some sort of answer we got about how well our intuition performed), but in highly novel situations or in situations where we receive no feedback on how well our intuition is performing, our intuitions may be highly inaccurate (even though we may not FEEL any less confident about our correctness).

This essay was first written on October 7, 2018, and first appeared on this site on December 31, 2021. 

Here are 24 different methods you could use to better understand the way humans work. In each case, I use “trying to figure out new things about post-traumatic stress disorder (PTSD)” as an example. What techniques am I leaving out here?

1. Papers: Reading academic studies (e.g., from social psychology, consumer psychology, cognitive psychology, neuroscience, etc.)

Example: reading the newest academic papers on treatments for PTSD.

2. Products: Carefully examining products and apps that have been successful (as well as, potentially, those that have failed) to understand what human needs they were or weren’t satisfying and why they succeeded or failed.

Example: looking at what mental health apps related to PTSD are popular in the app stores.

3. Outliers: Investigating extreme case studies of real-life events (e.g., case studies of people with extreme psychological conditions who may have been reported about in the news or true stories about what actually happened when people were placed into really extreme conditions).

Example: reading about cases of extreme cults and whether or not each cult produced PTSD-like symptoms in its members.

4. Society: Examining the large-scale trends and organization of societies, including potentially human cultures around the world (American culture, traditional cultures, etc.) and how they are similar or different, and which societal structures tend to be created and what the effects of these structures are, etc.

Example: investigating which sort of societies PTSD appears to be much more prevalent in.

5. Conversations: Talking to smart and insightful people about what they think is true of human psychology. You can also show them the models or theories you have so far to get their criticism and suggestions for improving them.

Example: asking really smart (non-experts) who had PTSD what their current theories about PTSD are.

6. Reading: Reading books, blog posts, media articles, etc., containing information from smart or insightful people about psychology (including potentially writing from life coaches, relationship experts, marketers, thoughtful bloggers, user experience designers, game makers, etc.)

Example: seeing what theories bloggers have proposed regarding PTSD that many academics may not have considered.

7. Experts: Reaching out to experts and talking to them to understand their models of psychology (e.g., psychologists, psychiatrists, game developers, etc.). You can also show them the models and theories you have so far to get their criticism and suggestions for improving them.

Example: talking to world-class experts in PTSD.

8. Textbooks: Reading academic textbooks about psychology, cognitive science, etc.

Example: Linehan’s textbook on DBT and thinking about possible applications of DBT in PTSD treatment.

9. Models: Looking at what models or frameworks of psychology others have created (e.g., in social psychology, theoretical cognitive neuroscience, gamification research, persuasion research, etc).

Example: looking at the model of PTSD that is used in Cognitive Behavioral Therapy treatment.

10. Intuition: Our own intuition for what’s true about humans (presumably based on our first-hand experience interacting with others, as well as our first-hand, ingrained experience of just being a human).

Example: We may have the intuition that after experiencing a traumatic event, people will tend to start having frequent disruptive thoughts about that event, even when doing unrelated things.

11. Interviews: talking to individuals or groups about their thought processes, feelings, behaviors, etc., to understand their individual psychology and then attempting to extrapolate that knowledge to humans more generally.

Example: interviewing 20 people with PTSD about what their experience of it is like and what does or does not seem to help them.

12. Application: attempting to put into practice a psychological theory (e.g., by coaching individuals using a psychological technique or by trying to build an app that implements it) and paying attention to the ways it seems to work or not work.

Example: creating a tool for people with PTSD and making it publicly available, then monitoring how people respond to it.

13. Anecdotes: psychologically surprising or interesting anecdotes that we’re confident actually happened (e.g., because they happened to us or were reported on by a trustworthy source) that may have a bearing on human psychology.

Example: a story a friend told you about how they (believe) they fully cured their own extreme PTSD in a matter of days.

14. Self-observation: careful (and honest) observation of our own thoughts/beliefs/behavior/emotions and internal processes.

Example: observing the mental processes in your own mind that seem to occur after a very upsetting event has happened.

15. Self-experimentation: simply trying techniques and paying close attention to what effects they appear to have and what it feels like to apply them.

Example: try the technique of writing down for a week every upsetting thought that you notice yourself having, as well as writing down everything you observe or learn about the experience of doing this.

16. Deduction: combining propositions we already believe to produce new propositions we didn’t realize before that may be true.

Example: knowing that women with PTSD are much more likely to have associated depression than men with PTSD, and knowing that depression is a strong risk factor for attempting suicide, we might predict that women with PTSD are more likely to attempt suicide than men with PTSD (even though they may not be more likely than men to actually commit suicide).

17. Experiments: Conduct your own online randomized controlled trials, surveys, longitudinal studies, or fMRI studies, etc.

Example: looking at which variables that don’t seem obviously PTSD related are, in fact, strong predictors of whether someone has PTSD.

18. Statistics: looking up known statistics about a phenomenon.

Example: looking up whether younger people or older people are known to be more likely to get PTSD in any given 12-month period.

19. Data: finding existing data sets (e.g., large longitudinal surveys or government data sets) and running your own statistical analyses to test hypotheses.

Example: measuring whether it is true that after someone gets a pet, they are more likely to recover from PTSD than similar people who do not get a pet. Running studies can also be a good way to search for new hypotheses. For instance, examining which of many variables in the data are most associated with a rapid recovery from PTSD.

20. Literature: looking at portrayals of psychology in literature, art, myths, stories, philosophical writings, etc.

Example: What hypotheses do Kafka’s novels give us about the nature of trauma?

21. Crowdsourcing: Request suggestions for hypotheses on a social media platform like Facebook or X or on a question-answer site like Quora.

Example: posting to Facebook asking people to suggest theories for why PTSD sometimes does and sometimes doesn’t happen after extreme trauma.

22. Social experiments: bringing a group of people together under a new set of chosen social rules or guidelines and observing what happens.

Example: getting a small group of people together with the idea that it will be a time to discuss traumas that members of the group have experienced.

23. Evolution: considering what possible evolutionary function different psychological phenomena might have.

Example: consider ways that aspects of PTSD responses might be evolutionarily adaptive.

24. Practitioners: examine how people “in the field” who benefit from being able to understand this aspect of psychology do their work.

For example, at veterans hospitals where rates of PTSD are high, investigate what techniques and approaches the therapists have developed to work successfully with their patients.

This piece was first written on November 16, 2017, and first appeared on my website on July 22, 2025.