Importance Hacking: a major (yet rarely-discussed) problem in science

December 19, 2022

I first published this post on the Clearer Thinking blog on December 19, 2022, and first cross-posted it to this site on January 21, 2023. You have probably heard the phrase "replication crisis." It refers to the grim fact that, in a number of fields of science, when researchers attempt to replicate previously published studies, they fairly often don't get the same results. The magnitude of the problem depends on the field, but in psychology, it seems that something like 40% of studies i...

How can we look at the same dataset and come to wildly different conclusions?

November 30, 2022

Recently, a study came out where 73 research teams independently analyzed the same data, all trying to test the same hypothesis. Seventy-one of the teams came up with numerical results across a total of 1,253 models. Across these 1,253 different ways of looking at the data, about 58% showed no effect, 17% showed a positive effect, and 25% showed a negative effect. But that's not even the oddest part. The oddest part is that despite a heroic attempt to do so, the study authors failed to...

Soldier Altruists vs. Scout Altruists

April 23, 2021

There is an important division between people who want to improve the world that few seem to be aware of. Inspired by Julia Galef's new book (The Scout Mindset), I'll call this division: Soldier Altruists vs. Scout Altruists. 1. Soldier Altruists think it's obvious how to improve the world and that we just need to execute those obvious steps. They see the barriers to a better world as: (i) not enough people taking action (e.g., due to ignorance, selfishness, or propaganda), and ...

It can be shockingly hard just to understand three variables

April 19, 2021

In science (and when developing hypotheses more generally), it is very common to come across situations where a variable of interest (let’s call this the dependent variable, “Y”) is strongly correlated with at least two other variables (let’s call them “A” and “B”). Here are some examples: If you’re a psychology researcher investigating possible causes of depression (Y), you may have trouble disentangling the effects of poor sleep quality (A) and anxiety (B), both of which tend to be corre...

What’s helpful and what’s unhelpful about postmodernism, critical theory, and their current intellectual offshoots?

March 29, 2021

More often than not, I find that postmodernist thought obscures rather than illuminates. But I also see useful elements in it. Here's my very un-postmodern attempt to "steel man" (i.e., find the value in) ideas related to postmodernism: 1. Narratives Serve Power - powerful groups do tend to have a substantial influence on narratives, beliefs, and what's "normal." Something "obvious" or "objective" or "a fact" may just (invisibly) be a part of the narrative you're immersed in and sub...

Disputes Over How to Use Statistics in the Real World

January 21, 2018

There is a surprising lack of consensus on how to do statistics, especially as applies to science. As the tool that underpins the scientific enterprise, you'd think we would have figured it out by now. You'd be wrong. The mathematical proofs are, of course, very rarely disputed. The use of mathematics is much more often disputed. Why do these disputes arise? I've observed five different types. Disputes in Applications of Statistics to Science (1) Disputes over philosophy: Exa...

The Brain of Theseus – a thought experiment

July 27, 2017

Here's my favorite philosophy of mind thought experiment that challenges pur view of personal identity or "self". It takes a while to explain but is quite a mind fuck, so bear with me. THE SETUP It feels, to nearly everyone, on a gut level, that I am 'me' and you are 'you', and consciousnesses are distinct from each other and easy to separate. Moreover, the vast majority of people accept that you 1 minute from now is still the same "YOU" in a meaningful sense as YOU right now; that ...

Testing Too Many Hypotheses

October 10, 2011

For each dataset, there is a limit to what we can use that dataset to test. Using the standard p-value based methods of science, the more hypotheses we check against the data, the more likely it will be that some of these checks give inaccurate conclusions. And this presents a big problem for the way science is practiced. Let's take an example to illustrate the principle. Suppose that you have information about 1000 people selected at random from the U.S. adult population. Your dataset includ...