I’d like to tell you about one of the biggest and most effortful studies on behavior change ever conducted and its strange findings (that may not even be obvious if you read the paper – I only realized its implications on my 2nd reading!)
Let’s begin: The study had 61,000 participants, all with gym memberships. 30 scientists worked in small teams to develop interventions, each aiming to increase gym attendance over 4-weeks. This resulted in 53 interventions to test. A promising setup to figu...
More
p-values
Demystifying p-values
There is a tremendous amount of confusion around what a p-value actually is, despite their widespread use in science. Here is my attempt to explain the concept of p-values concisely and clearly (including why they are useful and what often goes wrong with them).
— What's a p-value? —
If you run a study, then (all else equal, aside from rare edge cases) the lower the p-value, the lower the chance that your results are due to random chance or luck.
More precisely: a p-value is the probab...
More
Testing Too Many Hypotheses
For each dataset, there is a limit to what we can use that dataset to test. Using the standard p-value based methods of science, the more hypotheses we check against the data, the more likely it will be that some of these checks give inaccurate conclusions. And this presents a big problem for the way science is practiced.
Let's take an example to illustrate the principle. Suppose that you have information about 1000 people selected at random from the U.S. adult population. Your dataset includ...
More