Hi Spencer, thanks for the clarification.

Cheers!

Hi Ante. Sorry for the confusion – I was not saying that this isn’t a problem if (2) isn’t met; I was saying that it’s definitely a problem if (1) and (2) are met (if (1) and (2) -> problem, not: lack of (2) -> not problem). I updated the language for clarity.

This issue can be avoided by predicting the final outcome (not the change in that outcome) using the base value as an independent variable. Of course, this won’t settle WHY the outcome changed (e.g., you can’t tell if it’s regression to the mean, placebo, effect of the intevention itself, or something else), but it can tell that it did change (unlike the approach I critique in this article, where you predict the change in outcome and use the base value as an independent variable).

I came across a video of yours and have spent the last few hours going through your site with a lot of interest. I work on psychological interventions, and your post on using the baseline as a predictor of change hit a nerve, so let me think out loud for a moment.

The first thing it sharpened for me is the self selection angle. People tend to enrol in intervention studies at their worst, so even with no treatment at all you would expect their scores to come down at follow up, purely through regression to the mean. That alone should make us cautious about a lot of pre and post designs.

Where I wanted to push back was on your condition (2), that there is no mechanism by which a higher baseline drives a larger change. In symptom research it felt to me like something like that is almost always in play. Patients with higher initial scores have more room to move on a bounded scale, and there are real clinical reasons to think severity can moderate how much someone responds. So my first instinct was that condition (2) is routinely violated, and that this is a separate bias one would have to account for.

But following it through, I am no longer sure that helps the way I first thought. The measurement noise effect, the bounded scale room to move effect, and any genuine severity moderation all push in the same direction, higher baseline and bigger drop. So if I regress change on baseline, the coefficient is a blend of all three and I cannot tell which is which. If anything that seems to reinforce your point rather than complicate it.

So here is my actual question. Given that these sources all share a sign and stack on top of each other, is there any design or model that genuinely separates a real severity by treatment effect from the artefacts, short of a randomised control group with a baseline by treatment interaction term? Or is the honest answer that in observational intervention data they are simply not separable?

Thanks again for the post!

Best,
Ante