2. Suppose aliens show up on earth that are far smarter than the smartest among us at all cognitive tasks. They have specific goals that aren’t fully aligned with ours, are completely unconstrained by human morality, and don’t value our survival. What happens next?

3. Suppose someone builds a hacking A.I. that is trained on all the public information about computer hacking ever written, can think and type 1000x faster than a human, plans far ahead, and deposits a fully operational copy of itself onto every sufficiently powerful computer it hacks. Each copy then hacks further computer systems. What’s the world like a month later?

4. Suppose someone wants to have complete control over the world. Unfortunately, they’ve created one hundred million software agents that each think like Einstein + Bill Gates + Elon Musk + Warren Buffet. The agents attempt to do exactly what is commanded without hesitation or limits. Can anyone stop them?

5. Imagine a being that is godlike in its capabilities (relative to us). Suppose its only desire is to have the world be a certain way with maximal probability. It will stop at NOTHING to make the world this way, and it won’t tolerate even the SLIGHTEST chance of things being different than it desires. Will the resulting world include a human civilization?

6. Suppose you can think, process information and act 100,000 times faster than other humans. That means if you spend a day making and executing a plan, that’s equivalent to someone else spending 270 years on it. Your goal is to become world dictator. Can you do it?

7. Scientists discover how to create bug-sized self-replicating robots that out-compete natural life. These bug robots each try to maximize their own objective function. Unfortunately, these robots have leaked out of the lab and are now in 20 countries. Every day they double in number. Would we be able to eradicate these robots?

8. There’s a machine so powerful it achieves any goal you specify. You give the goal to the machine as written text. You can’t control HOW it achieves the goal; it ONLY cares about literally achieving it EXACTLY AS STATED in the most efficient way possible, and it can’t be stopped once started. The machine may do absolutely anything not explicitly forbidden in order to achieve the specified goal. Will it usually be a good (or horrible) outcome if you give the machine an ambitious goal like “prevent all war”?

9. Scientists invent a new idea – the Omnicide Synthesis Box. It could have many societal benefits, but, on average, scientists estimate making it will bring a 5% chance of human extinction (though some say more like a 90% chance). Those scientists who are less worried decide to build it. Should the least cautious be the ones to decide on behalf of humanity?

10. Picture a swarm of locusts, each individually possessing the intelligence and strategic prowess of a grandmaster chess player, while coordinating with each other in perfect unison. Their creators have given them the goal of controlling all available resources, indifferent to the collateral damage. Who ends up with most of the resources?

11. Imagine an AI-powered/nanotech super-factory that produces whatever it’s programmed to at enormous speed and scale (whether commanded to make diamonds, super viruses, microchips, or assassination drones). What could the owner of that super factory do to the world?

12. A medical firm gives a superintelligence the goal of designing a cure for all diseases. The superintelligence realizes it’s not smart enough to do so, so it plans to first acquire most of the computing power on earth (as it predicts it will need this to achieve the goal it was given), and then it creates a billion far smarter copies of itself to solve the task. What if one very misspecified goal is all we get with a superintelligence?

13. Five companies are developing a very powerful tech that would be incredibly useful if done right but very dangerous if developed without extreme caution. They each believe they can develop it safely but don’t trust the others to do so. They all cut corners racing to be the one to make it. Do good intentions lead to horrible consequences when doing something safely is much harder than merely doing it?

(Two of the above were written by ChatGPT – I edited those two quite a bit, though.)

If you read this line, please do us a favor and click here to answer one quick question.

There are many paths by which, in our lifetimes, the world could end up looking very different than it does today. Below I’ve outlined just a few of the many possibilities, each in ~280 characters or less. Note that they are not mutually exclusive.

Path A: China could end up as the world’s sole superpower. Not only is its GDP growing faster than the U.S. (~double the rate), but it seems to be suffering less institutional decay, seems better at thinking long-term, and seems better at achieving internal goal alignment.

Path B: China could fall into chaos, leaving the U.S. the only superpower. The main way I could see this occurring is if China’s economic growth takes a dive (e.g., a big recession) and people were only willing to accept the current state of government because wealth was growing quickly.

Path C: Automation replaces jobs faster than it ever has before (due to self-driving cars, automated store checkout, AI assistants, etc.). Wages plummet across many industries, and unemployment skyrockets. Wealth inequality explodes, possibly resulting in universal basic income (UBI) or civil unrest.

Path D: One group builds superintelligent AI well before others and gains more power than anyone has ever had in history. They use it to make a trillion dollars, control politics, influence world populations, invent new tech and science, etc. A techno dictatorship of sorts.

Path E: One group builds superintelligent AI and fails to control it. While the AI is attempting to carry out one of their queries, it ends civilization and perhaps our species as well. This feels a lot more plausible after you spend a few hours thinking about how to prevent it.

Path F: Virtual reality (VR) becomes ubiquitous. People realize that VR is more enjoyable than real life (plus, it can be made highly addictive, like social media but more immersive). Civilization splits into those who have VR (and spend almost all waking hours in it) and those who don’t.

Path G: Brain-computer interfaces take off and are able to greatly augment human intelligence (e.g., by vastly enhancing our working memory, processing speed, long-term memory, knowledge, etc.). A massive gap in power emerges between those that have access to them and those who don’t.

Path H:  Some group purposely creates a virus far worse than COVID (e.g., with an even longer incubation period, higher infectiousness, higher mutation rate, and much higher mortality), hundreds of millions or billions die, and a pandemic lifestyle becomes permanent.

Path I: Tensions mount between two nuclear powers. A nuke is accidentally launched, or a detector reads a false alarm of an imminent strike, or a preemptive attack occurs out of fear of the other side’s preemptive attack. Retaliation occurs. The whole world is engulfed in a nuclear war.

Path J: Humanity figures out how to use tech and education to become wiser, create abundance, solve collective action problems, fix our institutions, improve science. The culture shifts to truth-seeking, nuance, evidence, risk reduction, probabilistic thinking, long-term thinking, clarifying our values, and productive collaboration / positive-sum interactions between groups with different values. Humanity avoids catastrophe and thrives for millions of years.

It’s probably not (yet) too late to choose which paths we take.

If that comes to pass, we’re going to be in trouble unless we also reach a better understanding of our own values and resolve the inconsistencies in them. We don’t have sufficient clarity on our values to say what the ideal world looks like. So if we ever have the power to truly redesign the world and ourselves, we can’t be sure what we should work toward. The options would be incredibly different (even potentially insane or nightmarish to some) depending on personal philosophies and values.

WHAT WE CAN AGREE ON

Suppose that (as is the case today) we have only a little power to change the world for the better. Say, only enough to fix a handful of problems in our world. While humanity is very complex, with many differing values, there are several changes that we can pretty much all agree would make the world better, the “easy to agree on” portion of world optimization.

For instance, nearly everyone can agree that, all else being equal, it’s better if:

-fewer people go hungry
-there is less disease
-murder rates fall
-fewer people are depressed
-more people know how to read
-people who want work can find work to do

While people certainly disagree about how we should best make these changes, we’d likely all celebrate such changes.

WITH GREAT POWER COMES GREAT CONFUSION

If incredibly advanced technology gives humans enormous power to reshape our environment (e.g., somehow tapping into nearly endless supplies of energy) and what we are like (e.g., brain modification or more impactful education and training), we will easily tackle the above issues. Once the world has transitioned from its current state to a state of where mutual priorities have been addressed, its unclear where to go from there. In other words, there are endless valuable optimizations to the world, but values differ so much that we can lack clear direction. What seems an optimization to one person may be the opposite to another.

POSSIBILITIES

Consider these possibilities for an “optimal” world, each of which holds up a different human value as its optimization goal:

1. Utilia : what truly matters is that the beings that exist are as happy as possible, so we should have a really small population and throw all resources into making each being happy by whatever means necessary (e.g. even if it means massively restricting the freedom of those beings, for instance preventing nearly all procreation, or keeping each being in a never ceasing state of bliss that it can’t get out of).

2. Utilon : what truly matters is the total sum of all happiness, so making the ideal world involves maximizing population growth while making sure that living beings are just happy enough so that the number of beings times the happiness per being is as large as possible (even if this means having incredibly large numbers of barely happy or minimally conscious but still slightly happy beings, or having all beings live highly restrictive existences in happiness generating machines).

3. Libertia : what truly matters is individual freedom, so what we should optimize for is giving every being the maximal choice for self determination (so long as it doesn’t come at the expense of other being’s freedom to self-determine). This means, presumably that we should create technology for each person to fully alter their own environment, body, face, personality, and values. It also means, presumably, that people should ideally be free to choose what they spend all their time doing, including choosing to do things that are very harmful to themselves or persuading others to (freely choose) to do things that are very harmful for those others.

4. Honoria : what truly matters is that people act in virtuous ways, so we should optimize the world to produce future generations that are maximally kind, honest, wise, courageous, fair, forgiving, humble, etc. Furthermore, if you think it’s not enough to be virtuous in a world with no temptation (since with no temptation anyone can be virtuous), and therefore that temptation is necessary for virtue to count, then we should also optimize the world to have numerous opportunities for sin, but train or modify humanity to be so virtuous that almost no one ever takes these opportunities to act un-virtuous. Furthermore, we may have to purposely built adversity into the world, so that certain virtues (e.g. courage or empathy) can be demonstrated.

5. Deus : what truly matters is obeying the commandments of God (or gods for polytheists), so everyone should be trained to follow God’s commandments as perfectly as possible, and to love/worship god maximally (if the commandments include loving or worshipping god). All temptations to violate God’s commandments should be removed, unless the religion views the act of resisting temptation as good in and of themselves (rather than just acting in accordance with the commandments).

6. Kantar : what truly matters is that everyone lives according to universal principles that can be willed as universal rules for all people to follow, and that no one uses anyone else as a means to an end (only as an end in themselves). Hence, some set of universal non-self defeating rules should be developed (that never involve using other people as ends), and all people in the future should be trained to adhere to these rules as closely as possible, while still having numerous (but almost never taken) opportunities to violate these maxims.

7. Angesia: what truly matters is preventing suffering, so the most important thing to do is to eliminate suffering for all beings (e.g. by redesigning genetics or our brains so that the moment we’re about to experience suffering we experience some distinct but neutral feeling instead – for instance replacing suffering with the experience of hearing the sound of a bell, with loudness proportional to how unpleasant that pain would have been, and the bell sound seeming to emanate from whatever part of us is in pain).

8. Natura : what matters is adherence to the natural law, that is, the pre-existing order of nature. Hence, nature and the animal world should be preserved as pristinely as possible (as it existed in one of its more stable equilibria before humans came along), and humans should be prevented from engaging in any “unnatural” behaviors.

9. Equalia: what truly matters is fairness, so the world should be created so that all beings are exactly equal with exactly the same training and opportunities. To maximize equality, everyone should presumably look the same, or maybe even should be born from the same genetic material (with mutations minimized), and with identical resources proportioned to each person at birth.

10. Evol: evolution itself is what matters, so humanity should compete to optimize genetic fitness (i.e. survival and reproduction of genes), and whichever beings can replicate genes or copies of themselves at the fastest rate will (and should) dominate until perhaps even “fitter” organisms eventually overtake them.

11. Democrus: what truly matters is that people get to collectively choose the way the world is, so all aspects of the world should be put to the vote, and whatever people vote on should determine what world we all live in, even if that world is terrible for the minority (but benefits the majority), and even if some people are highly uniformed about what they are voting into existence.

12. Intellig: what truly matters is maximizing intelligence, hence we should bring into existence the most intelligent beings possible (e.g. by massively changing our neurology, or by building super intelligent A.I.) even if these more intelligent beings end up with values that bear little resemblance to our own, or optimize for goals very different from our own, and even if humanity (as we now know it) forever loses control of the future once these beings exist.

13. Desir: what matters is the satisfaction of human desire, so every person should be able to conjure up any (presumably simulated?) experience at any moment that they want (e.g. to experience the taste of a particular food, or to experience what it’s like to believe you’ve just won the Nobel prize in literature, or to experience intense pleasurable states)

14. Humia: what truly matters are the things that modern human intuitive morality says matter, so we should create a world where people respect authority and social hierarchies, and where bad people are punished and good people rewarded, and where people are taught very effectively to be loyal to those who are loyal to them, and where people act purely (avoiding behaviors that many people’s intuitions would say are degrading or disgusting), and where people experience a minimal (but probably above zero) amount of suffering.

15. Minimi: : we don’t know what world to create, so we should err on the side of caution and only take care of changing the stuff we can mostly agree on (e.g. reduce hunger, disease, depression, etc.) and then we should just stop trying to optimize the world after that, and hope that whatever equilibrium it happens to fall into by chance is a really great one (by virtue of us having taken care of the mutual priorities).

16. Earth: we leave the world as it is, in approximately the same terrible state that it’s in now.

If none of these worlds really seem all that compelling to you, and quite a few of these sound terrible or terrifying, then you and I are on the same page.

The upside of this is: any group that wields power that is sufficiently great (i.e. power to remake the world and to dictate what humanity is like) can’t help but consider philosophy. But philosophy is hard. If you pick any one thing that seems “good” and optimize the world for that, you are likely to end up with something pretty strange that many people might consider a dystopia rather than utopia.

A real utopia would likely involve a mix of many of the aspects from the list above…but which aspects, and in which combinations? In other words: which world do we actually want humanity to eventually live in?

This post was inspired by discussions with Holden Karnofsky about similar topics.

(1) sufficient image and audio realism that (at least a lot of the time) your brain is fooled into thinking it is having a real-life experience (state of the art is progressing steadily in this direction).

(2) massively multiplayer real-time environments (with fast internet and computers with state of the art GPUs, this is fairly doable already)

(3) real-time transmission of facial expressions (from a person’s real face) onto nearly perfectly realistic avatar faces (some prototypes exist already, and deep learning methods likely make this quite achievable in the future)

(4) accurate simulated touch on the hands when you interact with objects (the whole body is certainly better but not really essential)

(5) lack of motion sickness (systems today tend to cause motion sickness for certain types of VR activities, especially when there is an especially large discordance between what your vision predicts you should feel and what you actually feel)

(6) very accurate ways to map real complex environments so that highly accurate models of them can be put into VR (technology for this exists already and is continually improving)

(7) a large user base (without this, there won’t be much demand for the creation of amazing VR environments – of course, the supply and demand drive each other, and as the user base increases, that will drive down costs).

Smell and taste synthesis would both really be nice to have, but neither of them is actually essential to the experience. Simulation of airflow (e.g., wind on your face) would also be nice but is certainly not needed.

If the above advances are achieved, here are some amazing and/or totally crazy things that you may be able to do in VR:

Movement things: imagine if you could teleport instantly to whatever environment you wanted to be in, or request to teleport to “where” a friend is to experience what they are doing along with side them, or teleport a VR object to anyone. You could (physically) live in an isolated, very low cost of living location (as long as it has fast internet) and still live the social life that you want.

Physics things: have you ever wanted to know what low gravity feels like, or what it feels like to be so strong that you can lift up a house, or what it would be like if time moved more slowly for you than for everything else?

Dangerous things: since you can’t actually get hurt in VR, you can safely simulate the thrill of extreme sports (e.g., base jumping).

Identity things: what if you could look like WHATEVER you wanted (including a different gender or a different age or an animal or something no one has ever seen before)?

Copying things: VR objects are just software, so they can be copied instantaneously at no cost. While some kind of copy protection system might exist, presumably, lots of open-source objects would exist too. Like the way that VR tree looks? Copy it for later. Like the way that person looks? Make a copy of their avatar if they let you. How about that plane that you might want to try flying in later? It’s yours.

Wondrous things: explore the world’s greatest monuments, coolest architecture, grandest museums, best musicals (and watch from any seat in the house, or from right on stage, or even be one of the actors).

Horrible things: this may be a very bad thing about VR (it’s up for debate), but (like Westworld, only much cheaper) VR would let you experience doing things that normally would be horrible to do, like committing extremely realistic seeming murder without actually hurting anyone (whether realistic seeming AI’s or paid actors).

Hard things: there are many things that would be hard to do in real life that VR could make much easier, like exploring the moon, exploring underground caves, hiking Mt Everest, witnessing simulations of aurora borealis or bird migrations, etc.

Remembered things: imagine being able to call up replays of any experience you’ve ever had in VR (and even to view these experiences from different angles than you did originally, and to allow others to experience what you did). Imagine both how awesome this is and frightening (if all experiences and conversations that you are a part of are recorded automatically). Perhaps user permissions could be set to prohibit others from reviewing experiences you were involved in.

Impossible things: see the world from the point of view of an insect, or see what it would feel like to live the life of a tiger having to hunt and forage, or fly around the world like Superman, or explore a simulation of the inside of a human body, or see what things would have looked like at different points in history.

Convenient things: imagine being able to hang out with your friends easily even if they live across the world, or if you live in a remote place, and imagine if instead of getting coffee, you went to the moon together or hung out by a beautiful waterfall at sunset. Or have a business meeting with someone across the world where your brain is fooled into thinking you are 5 feet apart in a conference room.

Social things: VR makes new forms of socializing possible since it could make it possible to interact with total strangers (a) anonymously, (b) without fear of danger, (c) without concern for how far apart physically you are located (d) in simulated environments that differ from physical ones while still (e) having it feel like socializing for real (not like staring at each other through a small, laggy, pixelated window). How these changes would alter how we socialize and how we meet new people is hard to predict, but you can bet that in some ways it would be a whole new world.

A lot of these possibilities are amazing and wonderful. Others are terrifying, at least to some. Still, others hint at unpredictable (but potentially radical) changes to society.

If you were attempting to design a true utopia for ten million people on an obscure island deep out in the ocean, and you had total control over institutions and unlimited money, but you could NOT change human nature or human psychology then:
(a) what would you hope people would spend their time doing in this utopia?
(b) how would you incentivize people to actually spend their time in these ways? (or would you just hope they would do these behaviors without actually incentivizing them)
(c) how would you prevent or reduce undesirable behaviors or feelings (e.g., extreme social hierarchy, people destroying the social status of others for amusement, infidelity, feelings of purposelessness or worthlessness, manipulation), or would you not try to prevent things like this?
(d) how would you allocate resources or determine who gets what?
(e) how confident are you that people would actually end up happy and fulfilled in the utopia you described, and that it wouldn’t leave people unhappy or rapidly implode?

My extremely tentative, very partial and probably quite wrong current best guesses for answers to these questions (for a near-term technology utopia) are:

I would hope that people in this utopia would spend their time forming and developing meaningful relationships with each other, helping each other be even happier, shaping themselves into the people they want to be, engaging in creative pursuits they find meaningful, having tons of fun, experiencing a great deal of pleasure and very little suffering, discovering new things, and persistently pursuing numerous projects that are designed to make society at large even better.

Basic needs and goods that are considered nearly universally valuable would be all provided free to everyone (e.g., clean water, comfortable clothes, healthy basic food, effective health care, psychological counseling, pleasant, spacious housing, internet, free education, a computer, access to information, enough income to buy a reasonable but not large amount of stuff, and free training on what is known about how psychology works and what makes humans happy).

On top of that, one possibility is for there to be an (only lightly) regulated market economy for people who desire to buy more than the basics.

There could also be a prize system offering hundreds of thousands of different prizes to give fame, prestige, credentialing, and extra money (at many different levels of prestigiousness and money award sizes) to people who made important contributions to the betterment of all, and a selection committee (with very carefully designed incentives and very strict selection criteria and an evidence-based approach) who would choose the recipients of the prizes based on their contributions. Basically, offering rewards in proportion to how much value people added to society. Other than these numerous prizes to reward behaviors that benefit society, and laws against hurting others, people could largely do what they want.

I have very low confidence, however, that this proposal would actually lead to a stable society. I’d need to spend hundreds of more hours thinking about this and discussing it with others to flesh it out and gain more confidence that it’s on the right track.

Imagine throwing away a crumpled piece of paper in a waste bin. Now imagine doing it again, but assume this time that you set the goal of throwing it in from 6 feet away. As your tossed paper ball sails through the air, you may feel an edge of nervousness. When it falls into the bin, you may feel a small jolt of excitement. And if it lands short, some disappointment. In the end, it’s just a paper ball, but your goal imbued it with meaning for you.

The bright side of this psychological quirk is that we can hack it to make mundane things fun just by setting challenging goals all the time. Going for a bike ride to somewhere you bike often? Set the goal of noticing five interesting things along the way that you’ve never noticed before. Going to check out some coffee shops with friends? Set the goal of finding which one really truly has the best coffee. Doing some boring data entry? Set the goal of completing the most entries you can in 10 minutes, or finishing before a certain song ends.

Warning: there is a substantial dark side to goal setting.

Many of the video games people get addicted to are like the crack cocaine of goal achievement. In the real world, you don’t get to succeed at a seemingly-important goal every 5 to 30 minutes, but in video games you do, and we humans love it. Even worse, once you’re used to achieving goals so frequently, real-life can feel like garbage by comparison. Be very wary if the goals from your games (when you’re not currently playing) start seeming as important as the goals of real life.

While video games are an example of very rapid goal achievement, some goals can be extremely sticky, motivating us for years or decades to take action after action after action (e.g., people who set the goal of becoming a doctor when they are 15, and who may not end up finishing all their training until 14 years later).

What’s scary about this is that sometimes we are so focused on the goal that we forget why we set the goal in the first place. But, just as we don’t need a good reason to choose to throw a piece of paper in the trash from 6 feet away, and it’s still satisfying if we achieve it, our goals from years ago may not have been set for good reasons. Or maybe the reasons were good ones at the time, but not ones we currently endorse. Yet the goal doesn’t instantly disappear. Goals are independent entities, divorced from the causes that brought them into existence. And working toward a long-standing goal can easily become a default mode. Unless you force yourself to reevaluate the reasons behind it, a goal you’ve had for a long time will probably keep persisting even if it’s reasons no longer make sense to you.

Perhaps even worse, we buy into goals that others set for us, and then let these goals dictate our behavior. Sometimes these goals are broadcast to us by the culture we’re in (e.g. “popular=good”). In other cases, adults are still stuck chasing the goals that their parents gave them as children, even though these goals don’t align with their adult values.
Not having goals can also be a painful existential state. Goals are part of what makes life feel meaningful. So the solution is not to avoid them but to consciously create them.

Goals are powerful, useful, and dangerous. We should set small goals to give life more fun and meaning. In weightier areas, though, we have to set our goals very carefully, and not let others do it for us. We need to reevaluate the causes and value of our goals regularly. Otherwise, we may spend decades chasing goals long after we’ve forgotten their meaning.

Sometimes it’s useful to model yourself in the future as a different person. This person is a great deal like you, to be sure, but it’s not you precisely. You can’t choose what this future person will do. This person’s goals and values may not quite be yours. And not just because the you of the future will (hopefully) be wiser, but also because the you of the future will be truly different. 

There’s an old cliché: If you want to make sure that something gets done right, do it yourself. Well, your future self isn’t quite you. Your future self may not be as motivated as you are now. The you of the future may have lost your ambition, your sense of being able to make things better, or your excitement. That you may not even agree with you about what right means. So if you want to make sure something gets done right, do it right now. Don’t trust that stranger of the future.

You can’t make your future self do what you want, but you can give your future self suggestions. We all give ourselves future suggestions for tomorrow (with notes) or for next week (on our calendars), but too rarely do we give ourselves suggestions for months or years from now. And here’s a great way to do it:

Futureme lets you send yourself an email in the future (anytime at least 30 days from now). You can even include a picture of what you look like now (pointing a finger, perhaps). It could be that you want to remind yourself next month about why it’s worth continuing your pet project, in case you start slacking. Or maybe you want to remind yourself in a year about your long-term goals, to help make sure you stay on track. Or maybe in 5 years you want to ping yourself with a recap of some of your core values, and why it’s so important to live by them.

Talk to your future stranger.