Here’s hoping philosophers will make further progress on some of these!

1. The Trolley Problem: self-driving cars will be put in situations where there is a choice between sacrificing the passenger to save two pedestrians. Or they might have to choose between killing a 5-year-old pedestrian vs. killing a 70-year-old one.

2. The Hard Problem of Consciousness: every year, >100 billion mammals, birds, and fish live seemingly miserable lives on factory farms while ~1 trillion insects are raised yearly for food/animal feed. To what extent can insects/sardines/tuna/chickens/cows/pigs experience suffering?

3. Designing Utopia: if humanity survives long enough, and if technology grows exponentially in the long term, someday, we should have the technology to create something close to a paradise on earth. But what kind of paradise would we actually WANT to create? What should we be aiming for?

4. The Teletransporter: one-day technology may make it possible to upload or simulate a human brain on a computer. Cryonics contracts even let you specify if you would want this done to your frozen brain if/when it becomes feasible. But is that upload/simulation really YOU?

5. Uncertainty Uncertainties: some approaches to doing good allow expected value calculations from studies. Other ways require speculative arguments about the future. If, in both cases, we estimate a 10% chance of success, are these “10%”s equally good, even though one is much more uncertain?

6. Gender Disambiguation: there’s a battle over what precisely determines gender. Is it determined by your belief of what your gender is, a qualia of internally feeling like you are a specific gender, your soul (or some other non-physical property), your genes, your body, your gametes, your preferred social role, your presentation, your brain, the name we give to a choice each person makes, another permanent trait, another mutable trait, a cluster concept ambiguously referring to dozens of correlated traits, a meaningless concept, or something else? Maybe philosophers can at least narrow down this list of potential definitions by identifying that some are internally contradictory, incoherent, or at odds with what most people are trying to refer to.

7. The Nature of Intelligence: some think there’s an important sort of “intelligence” humans have that neural nets could never have (no matter how big, no matter how much data they were trained on). If so, what is the nature of this “intelligence”? If not, what are the implications for us, philosophy, and the future of humanity?

If you read this line, please do us a favor and click here to answer one quick question.

This piece was first written on August 10, 2021, and first appeared on this site on March 5, 2023.

If that comes to pass, we’re going to be in trouble unless we also reach a better understanding of our own values and resolve the inconsistencies in them. We don’t have sufficient clarity on our values to say what the ideal world looks like. So if we ever have the power to truly redesign the world and ourselves, we can’t be sure what we should work toward. The options would be incredibly different (even potentially insane or nightmarish to some) depending on personal philosophies and values.

WHAT WE CAN AGREE ON

Suppose that (as is the case today) we have only a little power to change the world for the better. Say, only enough to fix a handful of problems in our world. While humanity is very complex, with many differing values, there are several changes that we can pretty much all agree would make the world better, the “easy to agree on” portion of world optimization.

For instance, nearly everyone can agree that, all else being equal, it’s better if:

-fewer people go hungry
-there is less disease
-murder rates fall
-fewer people are depressed
-more people know how to read
-people who want work can find work to do

While people certainly disagree about how we should best make these changes, we’d likely all celebrate such changes.

WITH GREAT POWER COMES GREAT CONFUSION

If incredibly advanced technology gives humans enormous power to reshape our environment (e.g., somehow tapping into nearly endless supplies of energy) and what we are like (e.g., brain modification or more impactful education and training), we will easily tackle the above issues. Once the world has transitioned from its current state to a state of where mutual priorities have been addressed, its unclear where to go from there. In other words, there are endless valuable optimizations to the world, but values differ so much that we can lack clear direction. What seems an optimization to one person may be the opposite to another.

POSSIBILITIES

Consider these possibilities for an “optimal” world, each of which holds up a different human value as its optimization goal:

1. Utilia : what truly matters is that the beings that exist are as happy as possible, so we should have a really small population and throw all resources into making each being happy by whatever means necessary (e.g. even if it means massively restricting the freedom of those beings, for instance preventing nearly all procreation, or keeping each being in a never ceasing state of bliss that it can’t get out of).

2. Utilon : what truly matters is the total sum of all happiness, so making the ideal world involves maximizing population growth while making sure that living beings are just happy enough so that the number of beings times the happiness per being is as large as possible (even if this means having incredibly large numbers of barely happy or minimally conscious but still slightly happy beings, or having all beings live highly restrictive existences in happiness generating machines).

3. Libertia : what truly matters is individual freedom, so what we should optimize for is giving every being the maximal choice for self determination (so long as it doesn’t come at the expense of other being’s freedom to self-determine). This means, presumably that we should create technology for each person to fully alter their own environment, body, face, personality, and values. It also means, presumably, that people should ideally be free to choose what they spend all their time doing, including choosing to do things that are very harmful to themselves or persuading others to (freely choose) to do things that are very harmful for those others.

4. Honoria : what truly matters is that people act in virtuous ways, so we should optimize the world to produce future generations that are maximally kind, honest, wise, courageous, fair, forgiving, humble, etc. Furthermore, if you think it’s not enough to be virtuous in a world with no temptation (since with no temptation anyone can be virtuous), and therefore that temptation is necessary for virtue to count, then we should also optimize the world to have numerous opportunities for sin, but train or modify humanity to be so virtuous that almost no one ever takes these opportunities to act un-virtuous. Furthermore, we may have to purposely built adversity into the world, so that certain virtues (e.g. courage or empathy) can be demonstrated.

5. Deus : what truly matters is obeying the commandments of God (or gods for polytheists), so everyone should be trained to follow God’s commandments as perfectly as possible, and to love/worship god maximally (if the commandments include loving or worshipping god). All temptations to violate God’s commandments should be removed, unless the religion views the act of resisting temptation as good in and of themselves (rather than just acting in accordance with the commandments).

6. Kantar : what truly matters is that everyone lives according to universal principles that can be willed as universal rules for all people to follow, and that no one uses anyone else as a means to an end (only as an end in themselves). Hence, some set of universal non-self defeating rules should be developed (that never involve using other people as ends), and all people in the future should be trained to adhere to these rules as closely as possible, while still having numerous (but almost never taken) opportunities to violate these maxims.

7. Angesia: what truly matters is preventing suffering, so the most important thing to do is to eliminate suffering for all beings (e.g. by redesigning genetics or our brains so that the moment we’re about to experience suffering we experience some distinct but neutral feeling instead – for instance replacing suffering with the experience of hearing the sound of a bell, with loudness proportional to how unpleasant that pain would have been, and the bell sound seeming to emanate from whatever part of us is in pain).

8. Natura : what matters is adherence to the natural law, that is, the pre-existing order of nature. Hence, nature and the animal world should be preserved as pristinely as possible (as it existed in one of its more stable equilibria before humans came along), and humans should be prevented from engaging in any “unnatural” behaviors.

9. Equalia: what truly matters is fairness, so the world should be created so that all beings are exactly equal with exactly the same training and opportunities. To maximize equality, everyone should presumably look the same, or maybe even should be born from the same genetic material (with mutations minimized), and with identical resources proportioned to each person at birth.

10. Evol: evolution itself is what matters, so humanity should compete to optimize genetic fitness (i.e. survival and reproduction of genes), and whichever beings can replicate genes or copies of themselves at the fastest rate will (and should) dominate until perhaps even “fitter” organisms eventually overtake them.

11. Democrus: what truly matters is that people get to collectively choose the way the world is, so all aspects of the world should be put to the vote, and whatever people vote on should determine what world we all live in, even if that world is terrible for the minority (but benefits the majority), and even if some people are highly uniformed about what they are voting into existence.

12. Intellig: what truly matters is maximizing intelligence, hence we should bring into existence the most intelligent beings possible (e.g. by massively changing our neurology, or by building super intelligent A.I.) even if these more intelligent beings end up with values that bear little resemblance to our own, or optimize for goals very different from our own, and even if humanity (as we now know it) forever loses control of the future once these beings exist.

13. Desir: what matters is the satisfaction of human desire, so every person should be able to conjure up any (presumably simulated?) experience at any moment that they want (e.g. to experience the taste of a particular food, or to experience what it’s like to believe you’ve just won the Nobel prize in literature, or to experience intense pleasurable states)

14. Humia: what truly matters are the things that modern human intuitive morality says matter, so we should create a world where people respect authority and social hierarchies, and where bad people are punished and good people rewarded, and where people are taught very effectively to be loyal to those who are loyal to them, and where people act purely (avoiding behaviors that many people’s intuitions would say are degrading or disgusting), and where people experience a minimal (but probably above zero) amount of suffering.

15. Minimi: : we don’t know what world to create, so we should err on the side of caution and only take care of changing the stuff we can mostly agree on (e.g. reduce hunger, disease, depression, etc.) and then we should just stop trying to optimize the world after that, and hope that whatever equilibrium it happens to fall into by chance is a really great one (by virtue of us having taken care of the mutual priorities).

16. Earth: we leave the world as it is, in approximately the same terrible state that it’s in now.

If none of these worlds really seem all that compelling to you, and quite a few of these sound terrible or terrifying, then you and I are on the same page.

The upside of this is: any group that wields power that is sufficiently great (i.e. power to remake the world and to dictate what humanity is like) can’t help but consider philosophy. But philosophy is hard. If you pick any one thing that seems “good” and optimize the world for that, you are likely to end up with something pretty strange that many people might consider a dystopia rather than utopia.

A real utopia would likely involve a mix of many of the aspects from the list above…but which aspects, and in which combinations? In other words: which world do we actually want humanity to eventually live in?

This post was inspired by discussions with Holden Karnofsky about similar topics.

If you were attempting to design a true utopia for ten million people on an obscure island deep out in the ocean, and you had total control over institutions and unlimited money, but you could NOT change human nature or human psychology then:
(a) what would you hope people would spend their time doing in this utopia?
(b) how would you incentivize people to actually spend their time in these ways? (or would you just hope they would do these behaviors without actually incentivizing them)
(c) how would you prevent or reduce undesirable behaviors or feelings (e.g., extreme social hierarchy, people destroying the social status of others for amusement, infidelity, feelings of purposelessness or worthlessness, manipulation), or would you not try to prevent things like this?
(d) how would you allocate resources or determine who gets what?
(e) how confident are you that people would actually end up happy and fulfilled in the utopia you described, and that it wouldn’t leave people unhappy or rapidly implode?

My extremely tentative, very partial and probably quite wrong current best guesses for answers to these questions (for a near-term technology utopia) are:

I would hope that people in this utopia would spend their time forming and developing meaningful relationships with each other, helping each other be even happier, shaping themselves into the people they want to be, engaging in creative pursuits they find meaningful, having tons of fun, experiencing a great deal of pleasure and very little suffering, discovering new things, and persistently pursuing numerous projects that are designed to make society at large even better.

Basic needs and goods that are considered nearly universally valuable would be all provided free to everyone (e.g., clean water, comfortable clothes, healthy basic food, effective health care, psychological counseling, pleasant, spacious housing, internet, free education, a computer, access to information, enough income to buy a reasonable but not large amount of stuff, and free training on what is known about how psychology works and what makes humans happy).

On top of that, one possibility is for there to be an (only lightly) regulated market economy for people who desire to buy more than the basics.

There could also be a prize system offering hundreds of thousands of different prizes to give fame, prestige, credentialing, and extra money (at many different levels of prestigiousness and money award sizes) to people who made important contributions to the betterment of all, and a selection committee (with very carefully designed incentives and very strict selection criteria and an evidence-based approach) who would choose the recipients of the prizes based on their contributions. Basically, offering rewards in proportion to how much value people added to society. Other than these numerous prizes to reward behaviors that benefit society, and laws against hurting others, people could largely do what they want.

I have very low confidence, however, that this proposal would actually lead to a stable society. I’d need to spend hundreds of more hours thinking about this and discussing it with others to flesh it out and gain more confidence that it’s on the right track.