For instance, economists conduct a supply/demand or incentives-based analysis of the field of economics itself to understand why they focus on some areas and not others or why the field has become more math-heavy over time.

Psychologists can also study the psychology of academic psychologists to understand the underlying psychological drivers that determine which areas of study are popular or why the replication crisis occurred (from the perspective of the psychology of those who precipitated and enabled it).

Sociologists may also apply ethnographic methods to examine the institutions, practices, and self-concepts of sociologists.

After all, what could be more available to study and more at the top of your mind than your own group? And why not apply your field’s methods to your group since you are already applying them to everything else?

But in my experience, this kind of “Wesearch” – if you’ll allow me to coin a term – is quite rare and niche.

If I’m right about this, why would that be? I suspect part of the reason is that people want to see themselves as not being merely governed by simple forces.

It’s all fine and good to model other anonymous people as merely responding to incentives, being impacted by severe confirmation bias, mimicking each other’s behavior for social status, etc. But we don’t want to think of ourselves, and the colleagues we respect in that way. It feels reductionist (and inaccurate) to do so. Our colleagues might even feel insulted to be modeled in such a way. These are models only for everyone else.

An interesting exception, pointed out by a reader, is that academic psychologists often have run studies on graduate students (e.g., their mental health and other psychological challenges they face). But even still, that’s only an example of it studying one aspect of itself.

This piece was first written on May 23, 2025, and first appeared on my website on May 26, 2025.

A significant and pretty common problem I see when reading papers in social science (and psychology in particular) is that they present a fancy analysis but don’t show the results of what we have named the “Simplest Valid Analysis” – which is the simplest possible way of analyzing the data that is still a valid test of the hypothesis in question.

This creates two potentially serious problems that make me less confident in the reported results:

1. Fancy analyses impress people (including reviewers), but they are often harder to interpret than simple analyses. And it’s much less likely the reader really understands the fancy analysis, including its limitations, assumptions, and gotchas. So, the fancy analysis can easily be misinterpreted, and is sometimes even invalid for subtle reasons that reviewers, readers (and perhaps the researchers themselves) don’t realize. As a mathematician, I am deeply unimpressed when someone shows me a complex mathematical method when a simple one would have sufficed, but a lot of people fear or are impressed by fancy math, so complex analyses can be a shield that people hide behind.
2. Fancy analyses typically have more “researcher degrees of freedom.” This means that there is more wiggle room for researchers to choose an analysis that makes the results look the way the researcher would prefer they turn out. These choices can be all too easy to justify for many reasons including confirmation bias, wishful thinking, and a “publish or perish” mentality. In contrast, the Simplest Valid Analysis is often very constrained, with few (if any) choices left to the researcher. This makes it less prone to both unconscious and conscious biases.

When a paper doesn’t include the Simplest Valid Analysis, I think it is wise to downgrade your trust in the result at least a little bit. It doesn’t mean the results are wrong, but it does mean that they are harder to interpret.

I also think it’s fine and even good for researchers to include more sophisticated (valid) analyses and to explain why they believe those are better than the Simplest Valid Analysis, as long as the Simplest Valid Analysis is also included. Fancy methods sometimes are indeed better than simpler ones, but that’s not a good reason to exclude the simpler analysis.

Here are some real-world examples where I’ve seen a fancier analysis used while failing to report the Simplest Valid Analysis:

• Running a linear regression with lots of control variables when there is no need to control for all of these variables (or no need to control for more than one or two of the variables)
• Use of ANOVA with lots of variables when really the hypothesis only requires a simple comparison of two means
• Use of a custom statistical algorithm when a very simple standard algorithm can also test the hypothesis
• Use of fancy machine learning when simple regression algorithms may perform just as well
• Combining lots of tests into one using fancy methods rather than performing each test one at a time in a simple way

The problems that can occur when the results of Simplest Valid Analysis aren’t reported was one of the reasons that we decided to include a Clarity Criterion in our evaluation of studies for Transparent Replications. As part of evaluating a study’s Clarity, if it does not present the results of the Simplest Valid Analysis, we determine what that analysis would be, and pre-register and conduct the Simplest Valid Analysis on both the original data and the new data we collect for the replication. Usually it is fairly easy to determine what the Simplest Valid Analysis would be for a research question, but not always. When there are multiple analyses that could be used as the Simplest Valid Analysis, we select the one that we believe is most likely to be informative, and we select that analysis prior to running analyses on the original data and prior to collecting the replication data.

In my view, while it is very important that a study replicates, replication alone does not guarantee that a study’s results reflect something real in the world. For that to be the case, we also have to be confident that the results obtained are from valid tests of the hypotheses. One way to increase the likelihood of that being the case is to report the results from the Simplest Valid Analysis.

My advice is that, when you’re reading scientific results, look for the Simplest Valid Analysis, and if it’s not there, downgrade your trust in the results at least a little bit. If you’re a researcher, remember to report the Simplest Valid Analysis to help your work be trusted and to help avoid mistakes (I aspire always to do so, though there have likely been times I have forgotten to). And if you’re a peer reviewer or journal editor, ask authors to report the Simplest Valid Analysis in their papers in order to reduce the risk that the results have been misinterpreted.

• decide whether to marry your boyfriend or girlfriend
• choose between two job offers in different fields
• decide whether to finish your PhD program or drop out
• pick between two cities that you’re thinking of moving to

These kinds of decisions can be excruciatingly hard, and people often waffle in an unpleasant state for months trying to choose what to do. But fortunately, there are some strategies one can apply to help make the decision process shorter, and increase the chance that your ultimate decision does the best job of promoting what you value.

First, it’s important to know what not to do, as there are many pitfalls that arise in complicated decision-making situations.

Trap 1: Don’t just select the default. With some choices, there is a default that will occur if no choice is actively made. It is easy to select this default merely because you become paralyzed by the difficulty of the decision-making, or because you deliberate until the time runs out. Examples of this happening include: staying in your current city, merely because you couldn’t decide whether to move to another one, or maintaining your company’s existing business strategy merely because you couldn’t make the decision of which strategy to switch to. The default option is of course not always a bad one, but it is important to make sure not to choose it merely because it is the default.

One possible strategy for avoiding this default bias is to pre-commit to making a thought out decision by a specific date. You can recruit a friend or business colleague to hold you accountable. The deal will be that you have to send him a 400 word explanation of what you chose and why by the specified date, and if you don’t, you owe a certain amount of money for every day that you go over this deadline. Write a check and give it to that friend now, which he will cash and begin to spend on himself if you don’t meet the cutoff date. Just be sure to set the deadline far enough away to give yourself plenty of time to think about and research the problem thoroughly (keeping in mind that the human tendency is to underestimate how long projects will take).

Trap 2: Don’t just select whatever benefits you most right now. Frequently choices pit near term benefits against longer term ones. You could avoid talking to your friend who is angry at you, which benefits you short-term, or try to understand why he is angry, which will involve an awkward conversation and possibly getting yelled at. You could stay at your current job which you don’t like but don’t hate, or go through the frustrating and time-consuming process of cover letters, interviews and rejections, in the hope of landing a job you love.

Since we tend to be biased towards doing what is pleasant now, it’s important to correct for this when considering our choices. Near term benefits ideally should count for something, but how good you feel today from a decision, or even how good you feel for the rest of this month, is often of little consequence compared to how good your life will be for years to come as a result of your choice. So, for life changing decisions that are likely to have strong long-term effects on your happiness or other values, near term benefit shouldn’t factor into the decision-making process much. In these cases, it may be helpful to frame the decision mentally as “which of these choices will be best for me long-term”, rather than “which of these choices is best for me”. By specifically ignoring short-term benefit, the former phrasing of the question may help you be less biased by what is convenient or pleasant right now. Of course, in cases where the near term effects of a decision are a substantial part of the total effects, this method isn’t appropriate, and short-term benefits or losses need to be explicitly considered.

Trap 3: Don’t focus only on a few noticeable details. Sometimes the aspects of a situation that come to mind most easily or most vividly are not representative of the situation as a whole. For instance, if you are trying to decide whether to have children, what may come to mind is the image of reading a story to your children while they listen attentively with smiles on their faces. But someone else might get the image of changing a diaper while their child screams. While both of these images represent aspects of raising children, neither comes close to capturing the experience comprehensively. But depending on which of the two images comes to mind when you reflect on having children, you might end up making a different decision.

To be a good decision maker, it’s important that you not selectively focus on one feature of a situation, but rather view it more broadly. It can help to imagine the various aspects of the situation one by one, trying to mentally weight the frequently occurring and more likely aspects more heavily (for instance, by visualizing them for longer). When you think about having children, you should reflect both on your children smiling, and on them crying. You should also spend lots of  time focusing on the most common moments of child raising.

Trap 4: Don’t exaggerate the importance of a decision. People tend to overestimate the impact of both negative and positive events on their long-term happiness. Keeping this in mind can reduce the chance of obsessing about a decision, and can also help stop decision paralysis caused by the fear that a wrong decision would be devastating to your future. There is a decent chance that whichever option you choose, in the long-term you will be about as happy as you are now. So try not to give a decision more weight that in deserves. It’s especially easy to overestimate the importance of something merely because you happen to be reflecting on it. To help avoid these issues, try imagining what life would be like one year after the decision for each of the possible options you could take. How sure are you that you would feel significantly different on a day-to-day basis if you take option A rather than B?

For decisions that aren’t that important, it’s especially easy to make yourself less happy overall by worrying about them too much. A half hour of frustrated deliberation is probably not worth it when you’re deciding how to spend the $5 coupon you just got. Even if you make the “wrong” choice with the coupon, you’ll probably be happier on net without having wasted the time and given yourself the extra stress. And don’t forget that your time is worth money.

Now that we’ve considered traps to watch out for, let’s take a look at some strategies you can apply to help improve really important decisions in complex situations.

Strategy 1: Think about the advice you would give a friend. Sometimes it is helpful to forget for a moment that it is your own life you are deciding about. Imagine that a friend of yours (who you’d really like to help) is dealing with a decision very much like yours. Visualize her situation vividly, as though it were really occurring. Now imagine that this friend asks you which choice she should pick for herself. What would you tell her?

This method helps us detach ourselves from the emotions of the situation, which can sometimes prevent us from making the right choice. For instance, suppose that you’re trying to decide whether to break up with your boyfriend. Even though you enjoy being with him, you know he isn’t the sort of person you’d like to be with long-term. It can be a lot easier to give good advice to a friend who is in such a situation than to reflect in an unbiased fashion on what will be best for both you and your boyfriend in the long run. Fear of upsetting your boyfriend could stop you from breaking up with him now. But if your friend was in the same situation, you’d realize that this fear of hurting the boyfriend is not a good enough reason not to break up.

Strategy 2: List the pros and cons. Sometimes it takes careful thought to be aware of all the pluses and minuses of taking a particular course of action. Taking time to reflect on these and make a list of them can give you a significantly more complete understanding of the situation. Having a friend help is even better, since they may be able to think of advantages and disadvantages that you wouldn’t have. Once this list is complete, read it over a couple of times to help your brain process the entire list.

Keep in mind that when you come up with the pros and cons of multiple options, you need to have a baseline option to compare against, and this baseline needs to be the same for each of the options. So pick a baseline situation, and then what you will list as a pro and con will be determined with respect to this baseline. If your baseline includes an hour of free time per day, and one of your other options also offers an hour of free time per day, this is neither a pro or a con, it is merely neutral with respect to this baseline. If another option offers an hour and a half of free time a day though, then 30 minutes of extra free time each day would count as a pro for this option, since its offers 30 minutes more free time a day than the baseline.

Strategy 3: Do a full cost benefit analysis. If you want a formal procedure for making a really tough decision, try this one. Again come up with a list of pros and cons for each option, but this time also come up with three numbers for each of these pros and cons. The first number is a probability, which is your rough estimate of how likely this pro or con is to come into play at all if you choose the given option. The second, is an estimate of the benefit (or harm) towards your happiness or other goals this pro or con would cause on average on a weekly basis, assuming that the pro or con does in fact occur. A value of 1 would indicate a slight benefit on a weekly basis, whereas a score of 10 would indicate an extremely large weekly average benefit. A value of -1 would indicate a slight harm on a weekly basis, and a score of -10 wold indicate an extremely large harm weekly. The third and final number to estimate for each pro and con is how many weeks, on average, you can expect this benefit or harm to last, again assuming that the pro or con does occur to begin with. Once you’ve estimated these three numbers for a pro or con, multiply them together to get your total score for that pro or con. Now sum up these scores for all of them associated with a given option to get a total score of how good that option is, on net. Do this for each of the options, and compare the scores. If one of the options has a much higher total score than the other options, that is probably the one you should go with.

Why three numbers for each pro and con? Well, pros ands cons that are likely to occur should count more, ones that have a greater impact should count more, and those that last for longer should count more. So we consider (a) the chance of that pro or con occurs at all, (b) how good it will be on average during periods it does occur, and (c) how long (i.e. how many of those periods) you will get to enjoy that benefit for if it does in fact occur at all. Multiplying these three factors together for each pro and con, and then summing the result for all pros and cons corresponding to an option, is a way of estimating the expected value (i.e. total value, on average) of that option.

Strategy 4: Gather data about others who have made the same choices. Sometimes it is possible to figure out how well your choices worked for other people (or how similar situations worked out for you in the past), which can inform your decision now. If you think there are compelling reasons why your situation is different from these other people’s, you can at least start with the estimates derived from considering the experiences of others, and then adjust up or down based on these differences. For instance, suppose that you are thinking about quitting your steady job and trying to become a Broadway star. It shouldn’t be too difficult to determine that this goes poorly for most people who try it. If, however, you’re unusually good-looking, have objective evidence that you’re a fantastic singer even by Broadway standards, and happen to be friends with a lot of people who run theatre companies with shows on Broadway, this information should cause you to revise your chance of success upwards well above the typical levels. Note though that when considering what to do, people usually underestimate the relevance of how things typically turn out for others confronted with similar options. So it’s important to do this research when possible,  and not overestimate how special or unique your situation is. If things almost always go badly for people that select a certain choice, then they’ll most likely go badly if you make that choice as well.

Strategy 5: Visualize the options. Imagine, as vividly as you can, what life will be like if you choose each of the possible options. Try to make these visualizations as realistic as possible, gathering whatever information you can to make them realistic. Make sure these visualizations cover both the good and bad aspects of each choice. Spend more time focussing on the aspects of the options that will occur the most, since these tend to have a much greater impact on how good or bad a situation is overall. For instance, if you’re thinking about taking a job that is likely to be boring, don’t just visualize this boredom for a second and move on. Repeatedly imagine being bored sitting at your desk, since it is likely to be a major component of the experience, occurring a significance percentage of your days.  Once you’ve completed your visualizations, see which of the options you feel the best about. If there are some major points of uncertainty about what the options will be like, and these points of uncertainty would greatly influence how you feel about the options, visualize each possibility for each option, and write down a score from 1 to 10 representing how good you feel about it. Multiply each of these scores by how likely you estimate that possibility is to occur, and sum up these for each option to produce a total score for how good the option is.

Strategy 6: Persuade yourself with an essay. In important cases where there are only two options, try writing a 400 word essay as to why you should pick the first option rather than the second. Once that is done, write a 400 word essay about why you should choose the second option rather than the first. Read both of these over. This writing process can help you flesh out your thoughts, analyze the situation more completely, and discover more about how you feel about the various options.

Strategy 7: Poll good decision makers. Think of three or four people who are unusually good decision makers, and unusually careful at thinking through challenging problems. Now, explain to each of them the details of your situation, and ask what they think you should do in the circumstance. If they all agree with each other, that may be a compelling justification for going with their preferred option.

How can you choose between these seven methods? Well, if it’s a decision where you think emotion might be clouding your judgement, try thinking about the advice you would give a friend. If it’s a decision where there are many factors to consider, go ahead and list the pros and cons. If after doing that you still don’t know which option is better, do a full cost benefit analysis. If it’s a scenario where you can gather data about others who have made the same choices, try that. If you’d prefer to make an intuitive judgement over using a point system, go ahead and visualize the options. For decisions with just two options where you’re having trouble getting yourself to analyze the problem carefully, try to persuade yourself with an essay. If you know some careful thinkers who would be happy to try to help, then go ahead and poll good decision makers.

While there isn’t a single right answer as to how to approach making important, complex decisions, these strategies can help you formalize the process and avoid pitfalls, boosting the chance that you select the option that’s best overall.