Suppose that if we humans keep doing an activity, then we’ll have at least a P percent chance of causing the most horrible series of events so far in human history within 50 years.
How big does P need to be before it’s irresponsible to say, “Well, the horrible event is unlikely to happen, so let’s not worry about it”?
50% (1 in 2)?
5% (1 in 20)?
1% (1 in 100)?
0.1% (1 in 1,000)?
0.01%? (1 in 10,000)?
0.001%? (1 in 100,000)?
0.0001%? (1 in 1,000,000)?
To me, it seems bizarre to dismiss a threat that has even a 1 in 1000 chance of causing the worst events in human history (over the next 50 years) with an argument like “I think it’s unlikely to happen; therefore, let’s stop worrying about it.” Most people (quite rationally) decide it’s worth wearing a seatbelt with each drive, despite far less than a 1 in 1000 chance of a car accident on each trip. Yet, as a society, we often collectively go without our metaphorical seatbelts when it comes to societal risks. And with societal risks, unlike with seatbelts, we often can’t even really say what the probability of these risks is – some people have an intuition that they are very small, but many others believe the risk is high.
Of course, preparing for risks doesn’t mean we should stop everything else in the meantime; it just means that we should take collective action to reduce these risks as long as the cost per unit of risk reduction is not too high. And yet, we often fail to do so.
There are a number of activities humanity is doing right now that one could reasonably argue have a P of at least 1 in 1000 and perhaps much higher than that. For instance:
Our lack of strong controls on pathogen research enables the potential creation of engineered viruses or bacteria that could lead to terrible pandemics.
Our lack of safeguards on AI development leaves open the possibility of super-intelligent AI that we are unable to control or super-intelligent AI that gives one group unreasonable levels of power.
Our failure to create global coordination around climate change leaves open the possibility of catastrophic climate impacts.
The most popular argument given for ignoring these potential risks is that they seem (to the person who advocates ignoring them) unlikely to happen. While many disagree that these are unlikely, even if they are unlikely, they’d have to be really unlikely with a high degree of confidence for that to be sufficient grounds for not taking significant precautions.
This piece was first written on August 1, 2017, and first appeared on my website on June 11, 2025.
Comments