People often assume that the frequency with which a variable will take on different values (i.e., probability distribution) is likely to follow a bell curve (i.e., a normal distribution); this is often a mistake. Instead, consider these rules of thumb for deciding which probability distribution to use as a model in different situations:

(1) Binomial – when flipping coins

Number of occurrences out of a fixed number of tries: if the variable represents the number of times something happened out of a certain fixed number of trials that are at least mostly independent from each other (e.g., the number of times a team wins out of a fixed number of games, or the number of people on your mailing list that opened the last email you sent) then try modeling the probability of outcomes with a binomial distribution. Note that when the number of trials is large, and the rate of success is not too close to 0% or 100%, then this can be approximated with a normal distribution, but when the number of trials is small, or the rate of success is close to 0% or 100% a normal distribution may fit it very poorly.

(2) Poisson – when watching the clock

Number of occurrences in a fixed amount of time: if the variable represents the number of things of a certain type that happen within a fixed amount of time (e.g., 1 hour) and that time shouldn’t be affected by the time that’s elapsed since the last instance of the thing happening (e.g., the number of burglaries that will happen in a given city in a day, or the number of emails that you get on a Monday between 9 am, and 10 am) then try a Poisson distribution. Instead of fixed amounts of time, it can also be applied to fixed amounts of space or stuff, such as the number of mutations in a fixed-sized region of DNA once a given dose of radiation is applied.

(3) Exponential – when luck fades

Positive variables where larger values are always less likely than smaller values: if the variable is constrained to be positive and it’s reasonable to assume that higher values are always less likely than lower values, then the Exponential distribution may be a reasonable choice. It can also be applied to cases where the variable is constrained to be above a certain value rather than strictly positive (if we’re willing to shift the mean of the Exponential distribution). Exponential distributions are an especially good choice when modeling the time until an event if the estimated amount of time remaining until the event occurs is not affected by the amount of time elapsed so far (e.g., the amount of time until the next telephone call at a call center around a fixed time of day, say 2 pm). It is also the right choice when modeling how likely the thing is to occur when, with each amount of passing the time, the chance of the thing happening falls by a fixed percentage (e.g., the chance that someone has NOT gotten an injury after their nth day of riding a motorcycle if we can assume roughly a fixed chance of injury each day). Another application of the Exponential distribution is when you know a variable must be positive, but its maximum value is unlimited, and you know its mean value must be set in stone, but otherwise, you have the maximum uncertainty possible about outcomes of the distribution (i.e., you want to use the maximum entropy distribution for a known mean).

(4) Gamma – when anything could happen (but it’s got to be positive)

Positive variables that don’t have a clear maximum value – when the restriction of larger values being always less likely than smaller values is not necessarily valid (but you still have an unbounded variable that must be positive), then the Gamma distribution can be a reasonable choice. It generalizes the exponential distribution to a wider range of cases and also includes as a special case the famous chi-squared distribution. The Gamma distribution is useful for modeling things such as the total amount of rainfall in a certain country in a certain year (it must be a positive number, and we can’t say for sure that the maximum rain that has ever occurred is truly the maximum that could occur). Another example application would be the amount of time between one random car passing a certain point and the next car passing that point, which again must be positive but doesn’t have a clear maximum value.

(5) Normal – when things just add up

Sums of variables: if the variable can be thought of as a sum or average (or weighted average, where no one variable gets most of the weight) of other variables that aren’t that correlated to each other (e.g., human height, which can be approximated as being affected by a weighted average of different effects from different genes, or IQ test scores, which can be thought of as a sum of many different components of intelligence such as working memory, verbal comprehension, spatial reasoning, processing speed, practice taking such tests, focus, etc.) then try a Normal distribution (a.k.a. bell curve). Regarding human height, a Normal distribution fits it quite well if you consider adults, but even better if you just consider males or just consider females (since the male/female chromosome difference causes a relatively large change, which can be thought of as a variable with excessively large weight in the weighted average). A Normal distribution is also a good choice when you know that the center (e.g., mean or median) and width (e.g., standard deviation) are set in stone, but other than that, you have the maximum amount of uncertainty about what outcomes can occur (i.e., you want the maximum entropy distribution for a known mean and standard deviation), or when you’re trying to pick a reasonable prior distribution for the mean value of something (e.g., you want to express uncertainty about the mean of another probability distribution).

(6) Log-normal – when luck is compounding

Products of variables: if the variable can be thought of as a product of (as opposed to some of) positive variables that are at least mostly independent, or thought of as an accumulation of small percentage changes (from a positive starting point) due to different unrelated causes, then consider using a Log-normal distribution (a.k.a. Galton distribution). Examples where these assumptions seem to work reasonably well include, e.g., the income distribution of a large population (if we cut off the top 3% or so richest as that top tail may be better modeled as a Pareto distribution), or the distribution of populations of cities, or perhaps people’s total “success” in life by some fixed metric (excluding the very, very highest of achievers). The Log-normal distribution can also be a reasonable choice when a variable represents percentage changes in a thing from a fixed starting point of 1 (where 0 is the lowest possible value since it would indicate the thing is totally gone) when there is not a clear maximum (e.g., the distribution of returns for a diversified asset portfolio – though this distribution will leave out the extreme outlier tail risk caused by black swan events such as the world being hit by a huge Asteroid or a world-altering new technology). The log-normal distribution is also a reasonable choice when trying to set a prior distribution for a scale parameter (e.g., you need to use a probability distribution to model the uncertainty in the standard deviation of some other distribution) or if you expect the number of orders of magnitude of a thing to be well modeled by a normal distribution (so that the thing itself is the exponential of a normally distributed variable). When the mean is much larger than the standard deviation, a Log-normal distribution approximates a Normal distribution.

(7) Beta – when possibilities are penned in

Continuous variables constrained to be in a certain range: if the variable you’re trying to model the probability distribution of is a continuous variable that is only allowed to be in a bounded range (e.g., a corruption score that can be any value from 0 to 1 or a quality score in the range of 1 to 1000) consider using a Beta distribution (rescaled and/or shifted to cover the range of allowed values of your variable since by default the Beta distribution has the range 0 to 1). The Beta distribution works especially though well when the variable you’re trying to model the probability distribution of is itself a probability (which by definition is always going to be in the range of 0 to 1) or if you need a prior distribution for a variable that represents a probability. The Beta distribution can also work well when trying to model how likely the median value (or, more generally, the path percentile value) of a set of values is to take on different values if the original values are approximately uniformly distributed in the range 0 to 1. The commonly used uniform distribution, which assigns an equal likelihood to each value within the range 0 to 1 (and no chance to anything outside of that range), is a special case of the Beta distribution.

(8) Pareto – when outcomes are driven by outliers

80/20 type rules for positive variables: if you expect that the top X% of the outcomes will contribute Y% of the total, but ALSO that X% of those top X% will contribute Y% of the total of just that top group of outcomes, and X% of the top X% of the top X% will contribute Y% of the total of that top group of the top group, and so on and so forth, then consider using a Pareto distribution (a.k.a. power law). Many people believe this is the right model for the total value of startups or the amount of total funds raised by startups. The Pareto distribution also is a good choice when you are modeling the total outcome value of a thing that occurs over time, and a reasonable estimate for the value remaining to be added is proportional to the amount of value accumulated so far. So for instance, if a reasonable estimate for the final equity valuation of startups is 15% higher than the valuation at their last equity sale, then a Pareto distribution might be a reasonable choice to model equity valuation for startups. Or, to give another example, if the average time remaining for a project to finish is proportional to the amount of time elapsed so far, then the completion times of projects might follow a Pareto distribution. Pareto distributions can be good for modeling things where a lot of the total of all the outcomes is driven by really extreme outliers. Relatedly, Pareto distributions have the funny property that they can have infinite variance or even infinite mean for certain parameter values. Things in the real world don’t really seem to have infinite means or variances as far as we can tell, but in some cases, it could still be a reasonable model of real-world phenomena. As statistician George Box said, “all models are wrong, but some are useful.”

(9) Gumbel – when you’re studying the worst-case scenario

The maximum of a large number of values: if you are trying to model something characterized by a maximum (or minimum) of many values, such as the max amount of rainfall in a city / the max daily stock market decline / the maximum worldwide earthquake strength in any day over a ten year period, consider using a Gumbel distribution, Weibull distribution, or Fréchet distribution. Which to use depends on details about the “tails” of the distribution from which each individual value is drawn. See here for more details about “extreme value theory,” which characterizes what sort of distributions the maximum can have (when the maximum is taken over a large number of random values drawn from some distribution): https://en.wikipedia.org/wiki/Extreme_value_theory…

What are the possible explanations for (or causes of) this?

• A Causes B – Increasing A is a cause of increasing B but not the reverse. [e.g., more happiness causes more willpower]
• (2) B Causes A – Increasing B is a cause of increasing A but not the reverse. [e.g., more willpower causes more happiness]
• (3) A Causes B Causes A – Increasing A and B are both causes of increases of the other, leading to a positive feedback loop between them. [e.g., more willpower causes more happiness which causes more willpower which causes more happiness, etc.]
• (4) X Causes A and B – There exists at least one other variable X (potentially many more than one) such that increases (or decreases) in X lead simultaneously to increases in both A and B. [e.g., living in a stable environment and having a supportive romantic partner are both situations that increase happiness and are also both situations that increase willpower]
• (5) Non-linearity – A and B actually have a much stronger relationship than it appears (in fact they could be fully deterministically related) because correlation only captures the average extent to which A exceeding its mean coincides with B exceeding its mean, and can understate the strength of relationships that go both up and down or that go up in an inconsistent fashion (that is, it captures linear relationships, but can sometimes mask non-linear ones). [e.g., as willpower goes up happiness tends to go up as people become increasingly good at making good longer-term effective choices, but as willpower gets to extremely high levels, people start experiencing less happiness again because extreme levels of willpower are linked to reduced emotional responses of all kinds]
• (6a) Noise – The correlation is actually much larger or smaller than it seems as a result of sampling error or noise (e.g., the sample size is too low to measure it with reliability, and so it is really more like 0.32 plus or minus 0.3 and could realistically be near zero) [e.g., with a sample size of n=30, a measured correlation of 0.32 has an insanely wide 95th percentile confidence interval of -0.045 to 0.609]
• (6b) Too Many Tries – You found this correlation by looking at a long list of correlations that you calculated (and it stood out to you because it was one of the highest correlations), but there is actually not a significant correlation at all, it just looks that way because you tested so many hypotheses and each one had some chance of looking significant due to random fluctuations (so it’s unsurprising that one ended up being moderately large, and that’s the one you honed in on) [e.g., if you look at all pairs of correlations for 15 variables that leads to 105 distinct correlations, and with a sample size of 50 data points used to calculate each correlation, it is quite likely that you’d get a correlation of 0.32 or higher even if the true correlation between all pairs of variables was 0]
• (7) Outliers – There is an extreme outlier in the data (e.g., a transcription error or data corruption or disgruntled survey respondent), and without it, the calculated correlation would decrease or increase dramatically. [e.g., with n=100 data points, a crazy outlier pair of 30 for A and 1.35 for B when the rest of your values are between 0 and 1 can make the calculated correlation come out to 0.32 when the correlation calculated without that one outlier is actually -0.12]
• (8a) Selection Bias – Your data is a biased sample of the population that you actually care about due to unintentional selection effects in your sampling procedure, and so the 0.32 correlation, which you assumed applies to the broader population from which your data was drawn, is not actually representative of the population you are attempting to draw conclusions about. [e.g., you inadvertently surveyed only younger people, and in younger people, the correlation between willpower and happiness is positive, whereas in older people it is a negligible correlation, yet you’re trying to draw conclusions about the entire population which includes both young and old]
• (8b) Correlation From Conditioning – The variables A and B are actually uncorrelated if you had looked at them in the full population of interest, but you only sampled them in some subpopulation out of convenience, and A and B are both correlated with how likely you are to end up in that subpopulation, creating an observed correlation in the subpopulation that doesn’t exist in the full population [e.g., happiness and willpower are actually uncorrelated, but you only collected data on people at camp, and whether or not a person decides to go to camp is strongly positively predicted by happiness and strongly negatively predicted by willpower; therefore campers are likely to have high happiness or low willpower (though not both) creating a correlation between happiness and willpower among campers that doesn’t exist among non-campers]
• (9) The World Is Too Complex For Mere Mortals – Some mixture of (1), (2), (3), (4), (5) and (6a), (6b), (7), (8a), and (8b) are true.
• (10) Wrong Problem – Of the literally infinite variety of formulas that capture the extent to which A and B vary together, the correlation (i.e., the average of the product of standard deviations from the mean) is not the formula you are truly interested in, what you actually care about is, say, the median of the product of absolute deviations from the median, but you calculated the ordinary correlation because that’s all that you could think of to do, or that’s the only relevant function the software you used had available.
• (11) Undefined – A and B were generated from Cauchy distributions, and the true correlation between these two variables is mathematically undefined (much like how the Cauchy distribution has an undefined expected value), so your calculated value of 0.32 is a meaningless estimate of an undefined quantity.
• (12) Bug – there was a bug in the correlation calculation or data loading code, or data cleaning code and your software just happened to output 0.32, but that’s not actually the correlation value or a value of interest at all.
• (13) Wrong Calculation – You only thought you had run your correlation coefficient code, but you accidentally ran the wrong program (the one you wrote as a teen), and the number 0.32 was the fraction of baseball cards you had at that time that were in mint condition.
• (14) Memory Mistake – Your faulty memory only makes you think the correlation between A and B is 0.32, when in fact, the correlation number you saw on your computer screen (a mere 60 seconds ago) was 0.23 (thanks be to the unreliable three pounds of jello between your ears).
• (15) Dreaming – You are actually asleep right now (with your head slumped down on your keyword, and your knocked-over cup of tea dripping Earl Grey on the carpet), and you only dreamed that you calculated a 0.32 correlation, but when you wake up the correlation will turn out to be, say, -0.134 (or maybe you never collected the data in the first place and will get an F on your assignment).
• (16) God – God, it turns out, predetermined everything, and is, therefore, the one and only cause, so is the true (and only) explanation for this correlation of 0.32.
• (17) Solipsism – Metaphysical Solipsism turns out to be true, and the world has no independent existence outside of your mind, so in effect, you are the only cause, and therefore you are the cause of this experience of observation of 0.32 correlation (but what does it say about you that you would create this experience?)
• (18) Many Worlds – The Many World’s hypothesis is true, and the laws of physics don’t forbid any possible values of this particular correlation, hence for any number r within the range -1 to 1, there is a you who witnessed that correlation r, but most you’s most likely witnessed a correlation close to 0.32 (as evidenced by the fact that you witnessed it and that you are unlikely to have witnessed an unlikely event).
• (19) Simulation – The Simulation Hypothesis is true, you are part of a simulation, and the (simulated) world you live in was designed by conscious beings to have a 0.32 correlation between variables A and B (the true reason for which you will surely never understand, perhaps it has to do with getting published in a 5th dimension outer-reality academic journal on the niche topic of civilizational development in 4d space-time universes).
• (20) Relativism – Truth and meaning are relative and culturally determined, and the definition and usage of correlation, as well as the ritualistic “scientific” process of calculating it, and the belief that it should be calculated at all, are merely an artifact of your time and place, which props up the power of the current ruling elite, yet is so deeply rooted in your culture that you fail to see any other possibility.
• (21) Evil Demon – Descartes’ Evil Demon, as “clever and deceitful as he is powerful,” has been tricking you for the entirety of what you call your life, and all that you think you know of the external world is merely an illusion designed by this demon, including this godforsaken correlation.
• (22) Boltzmann Brains – The universe is infinite (in space, or time) and hence contains an infinite number of brains produced by the brief chance alignment of particles into structures capable of consciousness, and given that only a finite number of brains could evolve on planets, you have a 100% chance of being one of these randomly coalescing “Boltzmann brains,” and you just happen to be one that momentarily believes there is such a thing as correlation and that you measured a 0.32 correlation between A and B (in a moment you will pop back out of existence – bye!)
• (23) Contradiction – There is an as of yet undiscovered (but as the Incompleteness Theorem shows, impossible to rule out) contradiction in the axioms of mathematics as we know it, and a valid mathematical proof exists to show that A and B have any correlation including -1, Pi or 0.32.
• Or perhaps I’m overthinking this, and just: “as A goes up, B tends to go up a bit, on average.”