How big does P need to be before it’s irresponsible to say, “Well, the horrible event is unlikely to happen, so let’s not worry about it”?

50% (1 in 2)?
5% (1 in 20)?
1% (1 in 100)?
0.1% (1 in 1,000)?
0.01%? (1 in 10,000)?
0.001%? (1 in 100,000)?
0.0001%? (1 in 1,000,000)?

To me, it seems bizarre to dismiss a threat that has even a 1 in 1000 chance of causing the worst events in human history (over the next 50 years) with an argument like “I think it’s unlikely to happen; therefore, let’s stop worrying about it.” Most people (quite rationally) decide it’s worth wearing a seatbelt with each drive, despite far less than a 1 in 1000 chance of a car accident on each trip. Yet, as a society, we often collectively go without our metaphorical seatbelts when it comes to societal risks. And with societal risks, unlike with seatbelts, we often can’t even really say what the probability of these risks is – some people have an intuition that they are very small, but many others believe the risk is high.
 
Of course, preparing for risks doesn’t mean we should stop everything else in the meantime; it just means that we should take collective action to reduce these risks as long as the cost per unit of risk reduction is not too high. And yet, we often fail to do so.

There are a number of activities humanity is doing right now that one could reasonably argue have a P of at least 1 in 1000 and perhaps much higher than that. For instance:

Our lack of strong controls on pathogen research enables the potential creation of engineered viruses or bacteria that could lead to terrible pandemics.

Our lack of safeguards on AI development leaves open the possibility of super-intelligent AI that we are unable to control or super-intelligent AI that gives one group unreasonable levels of power.

Our failure to create global coordination around climate change leaves open the possibility of catastrophic climate impacts.

The most popular argument given for ignoring these potential risks is that they seem (to the person who advocates ignoring them) unlikely to happen. While many disagree that these are unlikely, even if they are unlikely, they’d have to be really unlikely with a high degree of confidence for that to be sufficient grounds for not taking significant precautions.

This piece was first written on August 1, 2017, and first appeared on my website on June 11, 2025.

If you were attempting to design a true utopia for ten million people on an obscure island deep out in the ocean, and you had total control over institutions and unlimited money, but you could NOT change human nature or human psychology then:
(a) what would you hope people would spend their time doing in this utopia?
(b) how would you incentivize people to actually spend their time in these ways? (or would you just hope they would do these behaviors without actually incentivizing them)
(c) how would you prevent or reduce undesirable behaviors or feelings (e.g., extreme social hierarchy, people destroying the social status of others for amusement, infidelity, feelings of purposelessness or worthlessness, manipulation), or would you not try to prevent things like this?
(d) how would you allocate resources or determine who gets what?
(e) how confident are you that people would actually end up happy and fulfilled in the utopia you described, and that it wouldn’t leave people unhappy or rapidly implode?

My extremely tentative, very partial and probably quite wrong current best guesses for answers to these questions (for a near-term technology utopia) are:

I would hope that people in this utopia would spend their time forming and developing meaningful relationships with each other, helping each other be even happier, shaping themselves into the people they want to be, engaging in creative pursuits they find meaningful, having tons of fun, experiencing a great deal of pleasure and very little suffering, discovering new things, and persistently pursuing numerous projects that are designed to make society at large even better.

Basic needs and goods that are considered nearly universally valuable would be all provided free to everyone (e.g., clean water, comfortable clothes, healthy basic food, effective health care, psychological counseling, pleasant, spacious housing, internet, free education, a computer, access to information, enough income to buy a reasonable but not large amount of stuff, and free training on what is known about how psychology works and what makes humans happy).

On top of that, one possibility is for there to be an (only lightly) regulated market economy for people who desire to buy more than the basics.

There could also be a prize system offering hundreds of thousands of different prizes to give fame, prestige, credentialing, and extra money (at many different levels of prestigiousness and money award sizes) to people who made important contributions to the betterment of all, and a selection committee (with very carefully designed incentives and very strict selection criteria and an evidence-based approach) who would choose the recipients of the prizes based on their contributions. Basically, offering rewards in proportion to how much value people added to society. Other than these numerous prizes to reward behaviors that benefit society, and laws against hurting others, people could largely do what they want.

I have very low confidence, however, that this proposal would actually lead to a stable society. I’d need to spend hundreds of more hours thinking about this and discussing it with others to flesh it out and gain more confidence that it’s on the right track.

The space of possible ideas is ABSURDLY, almost UNBELIEVABLY large. If we thought about a different idea every second for our entire lives, we wouldn’t begin to scratch the surface.

As a simple example, let’s consider the number of two-player competitive games played on an 8×8 chessboard, where each player starts with 16 pieces and each piece has a pre-determined fixed rule for how it moves across the board and captures other pieces. There are far more than a quintillion (i.e., 1,000,000,000,000,000,000) such games, and while many of these games are less than entertaining, there will be games in this set that are far better than chess along any dimension of gaming that you should care to consider. In other words, chess is our limited human attempt to mine a tiny corner of game space.

Of course, this type of 8×8 board game is just a tiny subset of all types of games, which is, in itself, a minuscule subset of all types of ideas.

This VASTNESS of idea space is why I find it so valuable when a person or book mentions a powerful, plausible idea that I’ve never heard before, even if no evidence is provided for why I should believe the idea.

The chance that you would independently have thought up an idea that someone tells you about is vanishingly small (even if you feel like you COULD have thought of it). The act of raising a powerful, plausible hypothesis to your awareness can be hugely valuable. Mere identification of a point in idea-space can involve a ton of work, even if it doesn’t seem like it.

Of course, we can then do our own vetting of ideas, even if the people who bring them to our attention don’t give us strong reasons to believe them. For instance, we can check them against facts that we already have strong reasons to believe. And we can ask ourselves, “does this idea contradict the evidence we have, or line up nicely with it? Does it explain things that confused us before and gel with other things we know to be true?” If we’re serious about testing the idea, we can even go out and gather new evidence about whether it’s true.

Some great ideas seem obviously true once you hear them. In fact, I’m a really big fan of trying to state the obvious; but not just any obvious things. The obvious things that people have likely not yet consciously considered. That is, what I like to call the “obvious and unsaid.” It bothers me when people dismiss obvious but rarely encountered ideas on the grounds that they seem obvious once you hear them because obviousness can be an asset rather than a liability.

These obviously true ideas can be especially valuable because they are easy to vet as being correct, yet due to the vastness of idea-space, they very easily could never have occurred to you before. So, stating obvious ideas that are rarely thought about, but which have important implications, can be an extremely efficient way to transmit value to others.

On a related point, when people say “everything’s already been done” or “there are no new ideas left,” the only way these statements could be sensible is if we interpret them as claims about humans rather than claims about ideas. For instance, as a claim that human minds are so similar to each other that, in the VAST space of possible ideas, we’re confined to a sad, well-trodden, little corner of derivative pseudo creativity.

However, I’m much more optimistic than this: while much is derivative, I think there are new ideas all over the place that we can learn to look for. As one person finds a new idea in a corner over there, and they share it over here, human knowledge advances. What’s more, I think that seeing new ideas as “derivative” is kind of missing the point.

Yes, good new ideas are often built from pieces of old ideas, but there’s nothing bad about that. Bricks are the building blocks of houses, and ideas are the building blocks of ideas.