2. Suppose aliens show up on earth that are far smarter than the smartest among us at all cognitive tasks. They have specific goals that aren’t fully aligned with ours, are completely unconstrained by human morality, and don’t value our survival. What happens next?

3. Suppose someone builds a hacking A.I. that is trained on all the public information about computer hacking ever written, can think and type 1000x faster than a human, plans far ahead, and deposits a fully operational copy of itself onto every sufficiently powerful computer it hacks. Each copy then hacks further computer systems. What’s the world like a month later?

4. Suppose someone wants to have complete control over the world. Unfortunately, they’ve created one hundred million software agents that each think like Einstein + Bill Gates + Elon Musk + Warren Buffet. The agents attempt to do exactly what is commanded without hesitation or limits. Can anyone stop them?

5. Imagine a being that is godlike in its capabilities (relative to us). Suppose its only desire is to have the world be a certain way with maximal probability. It will stop at NOTHING to make the world this way, and it won’t tolerate even the SLIGHTEST chance of things being different than it desires. Will the resulting world include a human civilization?

6. Suppose you can think, process information and act 100,000 times faster than other humans. That means if you spend a day making and executing a plan, that’s equivalent to someone else spending 270 years on it. Your goal is to become world dictator. Can you do it?

7. Scientists discover how to create bug-sized self-replicating robots that out-compete natural life. These bug robots each try to maximize their own objective function. Unfortunately, these robots have leaked out of the lab and are now in 20 countries. Every day they double in number. Would we be able to eradicate these robots?

8. There’s a machine so powerful it achieves any goal you specify. You give the goal to the machine as written text. You can’t control HOW it achieves the goal; it ONLY cares about literally achieving it EXACTLY AS STATED in the most efficient way possible, and it can’t be stopped once started. The machine may do absolutely anything not explicitly forbidden in order to achieve the specified goal. Will it usually be a good (or horrible) outcome if you give the machine an ambitious goal like “prevent all war”?

9. Scientists invent a new idea – the Omnicide Synthesis Box. It could have many societal benefits, but, on average, scientists estimate making it will bring a 5% chance of human extinction (though some say more like a 90% chance). Those scientists who are less worried decide to build it. Should the least cautious be the ones to decide on behalf of humanity?

10. Picture a swarm of locusts, each individually possessing the intelligence and strategic prowess of a grandmaster chess player, while coordinating with each other in perfect unison. Their creators have given them the goal of controlling all available resources, indifferent to the collateral damage. Who ends up with most of the resources?

11. Imagine an AI-powered/nanotech super-factory that produces whatever it’s programmed to at enormous speed and scale (whether commanded to make diamonds, super viruses, microchips, or assassination drones). What could the owner of that super factory do to the world?

12. A medical firm gives a superintelligence the goal of designing a cure for all diseases. The superintelligence realizes it’s not smart enough to do so, so it plans to first acquire most of the computing power on earth (as it predicts it will need this to achieve the goal it was given), and then it creates a billion far smarter copies of itself to solve the task. What if one very misspecified goal is all we get with a superintelligence?

13. Five companies are developing a very powerful tech that would be incredibly useful if done right but very dangerous if developed without extreme caution. They each believe they can develop it safely but don’t trust the others to do so. They all cut corners racing to be the one to make it. Do good intentions lead to horrible consequences when doing something safely is much harder than merely doing it?

(Two of the above were written by ChatGPT – I edited those two quite a bit, though.)

If you read this line, please do us a favor and click here to answer one quick question.

If you were attempting to design a true utopia for ten million people on an obscure island deep out in the ocean, and you had total control over institutions and unlimited money, but you could NOT change human nature or human psychology then:
(a) what would you hope people would spend their time doing in this utopia?
(b) how would you incentivize people to actually spend their time in these ways? (or would you just hope they would do these behaviors without actually incentivizing them)
(c) how would you prevent or reduce undesirable behaviors or feelings (e.g., extreme social hierarchy, people destroying the social status of others for amusement, infidelity, feelings of purposelessness or worthlessness, manipulation), or would you not try to prevent things like this?
(d) how would you allocate resources or determine who gets what?
(e) how confident are you that people would actually end up happy and fulfilled in the utopia you described, and that it wouldn’t leave people unhappy or rapidly implode?

My extremely tentative, very partial and probably quite wrong current best guesses for answers to these questions (for a near-term technology utopia) are:

I would hope that people in this utopia would spend their time forming and developing meaningful relationships with each other, helping each other be even happier, shaping themselves into the people they want to be, engaging in creative pursuits they find meaningful, having tons of fun, experiencing a great deal of pleasure and very little suffering, discovering new things, and persistently pursuing numerous projects that are designed to make society at large even better.

Basic needs and goods that are considered nearly universally valuable would be all provided free to everyone (e.g., clean water, comfortable clothes, healthy basic food, effective health care, psychological counseling, pleasant, spacious housing, internet, free education, a computer, access to information, enough income to buy a reasonable but not large amount of stuff, and free training on what is known about how psychology works and what makes humans happy).

On top of that, one possibility is for there to be an (only lightly) regulated market economy for people who desire to buy more than the basics.

There could also be a prize system offering hundreds of thousands of different prizes to give fame, prestige, credentialing, and extra money (at many different levels of prestigiousness and money award sizes) to people who made important contributions to the betterment of all, and a selection committee (with very carefully designed incentives and very strict selection criteria and an evidence-based approach) who would choose the recipients of the prizes based on their contributions. Basically, offering rewards in proportion to how much value people added to society. Other than these numerous prizes to reward behaviors that benefit society, and laws against hurting others, people could largely do what they want.

I have very low confidence, however, that this proposal would actually lead to a stable society. I’d need to spend hundreds of more hours thinking about this and discussing it with others to flesh it out and gain more confidence that it’s on the right track.