Testing a Theory Without an Experiment

You don’t need to run an experiment to perform a valid test of one of your theories or hypotheses (whether informal or scientific). There is a technique, which I’ll describe below, that can be far faster, and is used a lot less than it should be (especially when trying to test a theory in science, where it could save you an month long experiment, but also, with informal theories in daily life). I aspire to use this approach significantly more often than I do now.

How to Test a Theory Without Doing An Experiment:

Think of as many things as you can that are predicted by your theory, and try to identify at least one prediction that has the following four properties:

(1) your theory implies it is very likely to be true.

(2) it seems very unlikely to be true if your theory is false (i.e., there is no reason you can see to make that prediction other than if you believe in your theory). You may want to ask others if they agree that the prediction seems very unlikely to be true from the perspective of those who don’t believe your theory; otherwise, you might be deceiving yourself.

(3) you don’t already know the answer to whether it is actually true or not (you are merely guessing it is true because it is implied by your theory). This is important because if you already know the answer, you may be tricking yourself one way or another (e.g., by having already used this evidence in the construction of your theory, or by choosing this particular test because on some level you know it will make your theory look good).

(4) you can reliably look up whether that prediction is true or not. Then all you have to do is look up whether the prediction is true!

If it turns out to be true, that provides evidence to support your theory. More specifically, the more likely that true result is given your theory, and the less likely that true result is (to someone who doesn’t know your theory or if your theory is invalid), the more evidence the test provides. To explore the math for only a moment: that test of your theory should change your prior odds (i.e., your estimated probability that your theory is true, divided by the probability that it is not true) by using what’s known as the “Bayes factor,” namely, your estimation probability. Looking up the answer to that question would yield the result it did, if your theory is true divided by the probability it would yield that result if your theory is not true.

In other words:

odds your theory is true = ( probability of that result if your theory is true ÷ probability of that result if your theory is false) × prior odds your theory is true from before the test

Odds can be confusing to think about (remember that it is the probability that something is true divided by the probability that it is false). For instance, odds of 2 would mean that a thing is twice as likely to be the case then it is to not be the case.

It’s worth noting that the above procedure can work really effectively to test your hypotheses for yourself, but it’s NOT a good way to provide evidence that your theory is true to other people that don’t trust you (i.e., your honest pursuit of truth or your competence). The reason is that it’s easy to claim you conducted this fair test of your theory, when in fact, you already knew the answer to how the test would turn out (and therefore were incentivized to choose this particular test of your theory rather than another). Or perhaps you interpreted your theory in a weird way or tweaked it after the fact to make it look like it correctly predicted the outcome when it actually didn’t. Or maybe you screwed up one of the steps accidentally. In other words, the evidence this procedure produces is only trustworthy if you really trust the rigor and character of the person who produced the evidence.

This is why experiments are, at least in theory (but unfortunately too often not in practice), a better way to gather evidence when you need to communicate those results to others who don’t trust you, especially if you pre-register your study and analysis plan, and if you release your materials and the data, you collect.

If you’re interested in the concepts underpinning the approach I’ve described here, you may want to check out our program on Bayesian thinking (which you may or may not be happy to know uses almost no math):

http://programs.clearerthinking.org/question_of_evidence.ht…


  

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *