I’ve seen seven main viewpoints on AI and the future from those who spend a lot of time thinking about it:

(1) Superintelligence Doomers – they believe we are likely to build AI that’s superintelligent (i.e., that surpasses human intelligence in all respects) and that once we do, it will kill or enslave humanity.

See: Eliezer Yudkowsky

“The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else.”

(2) AI Corrosionists – they believe AI, while it could be beneficial, has a very substantial likelihood of making things far worse. Unlike doomers, they see most risk coming from other processes rather than a sudden AI takeover or annihilation: for instance, it could be that humans lose control over the future slowly (by ceding more and more control over to AIs that are optimizing for things that are different than human values), or it could be that AI de-stabilized aspects of society that make other large risks (like nuclear war) more likely

See: Paul Christiano

“If you imagine a society in which almost all of the work is being done by these inhuman systems who want something that’s significantly at cross purposes, it’s possible to have social arrangements in which their desires are thwarted, but you’ve kind of set up a really bad position. And I think the best guess would be that what happens will not be what the humans want to happen, but what the systems who greatly outnumber us want to happen.”

(3) Near Risk Doomers – they believe AI will have very bad effects (e.g., increasing unfairness, authoritarianism, climate change, unemployment, or concentration of power) but that we’re not going to build superintelligence anytime soon.

See: Cathy O’Neil

“The math-powered applications powering the data economy were based on choices made by fallible human beings. Some of these choices were no doubt made with the best intentions. Nevertheless, many of these models encoded human prejudice, misunderstanding, and bias into the software systems that increasingly managed our lives. Like gods, these mathematical models were opaque, their workings invisible to all but the highest priests in their domain: mathematicians and computer scientists. Their verdicts, even when wrong or harmful, were beyond dispute or appeal. And they tended to punish the poor and the oppressed in our society, while making the rich richer.”

(4) Current Paradigm Doubter – they believe the current AI paradigm is overhyped and isn’t going to change things all that much (some of them see the current paradigm as net negative; others see it as positive but only as one useful and overhyped tech among many other useful technologies). Some, like Marcus, hope that future paradigms might be more trustworthy and beneficial.

See: Gary Marcus

“What a strange world… All the major AI companies spending billions producing almost exactly the same results using almost exactly the same data using almost exactly the same technology, all flawed in almost exactly the same ways. Historians gonna be scratching their heads…The only way we will move significantly forward is to develop new architectures—likely neurosymbolic—that are less tied to the idiosyncrasies of specific training set.”

(5) AI Stewardists – they believe AI advancement is important to humanity’s future, but it’s not necessarily going to be good or bad: it will be what we make of it, so it should be developed thoughtfully based on what we want to achieve.

See: Kevin Kelly

“AI could just as well stand for ‘alien intelligence.’ We have no certainty we’ll contact extraterrestrial beings in the next 200 years, but we have almost 100 percent certainty that we’ll manufacture an alien intelligence by then. When we face these synthetic aliens, we’ll encounter the same benefits and challenges that we expect from contact with ET. They will force us to reevaluate our roles, our beliefs, our goals, our identity. What are humans for?”

(6) Near Benefit Boosters – they believe AI will be very useful, impactful and important as a technology, and they also believe superintelligence is a ridiculous thing to worry about.

See: Yann LeCun

“AI is intrinsically good, because the effect of AI is to make people smarter….AI is an amplifier of human intelligence and when people are smarter, better things happen: people are more productive, happier and the economy thrives.”

(7) Superintelligence Boosters – they believe we are likely to build superintelligent AI and that it will usher in an incredible and positive new era.

See: Ray Kurzweil

“By the time of the Singularity, there won’t be a distinction between humans and technology. This is not because humans will have become what we think of as machines today, but rather machines will have progressed to be like humans and beyond. Technology will be the metaphorical opposable thumb that enables our next step in evolution.”

How do we best organize the positions on risks from AI?

I think the two most informative spectrums to think about are

(1) How *substantial* the near-term impacts of AI are expected to be

and

(2) Whether the effects of AI are likely to be *good* or *bad*

The image shown is my attempt to place these different thinkers who talk about AI on this two-axis system. Though note that this just reflects my best guesses, some of these thinkers have very nuanced views, and I’m not certain that I’m placing each of them in the right place.

This piece was first written on July 27, 2024, and first appeared on my website on August 4, 2024.

But there is also something soul-crushing about it. People spend decades learning a craft and then see an A.I. make something in a few seconds or minutes that other people see as comparable to their own work.
And it will be widely misused: to create ripoffs of copyrighted works without giving credit, to generate billions of pages of low-quality content in order to game search engines, to make hyper-personalized spam and commit phishing and fraud, to mislead the public with misinformation and customized persuasion and outrage bait, and to create even more intense social media and video addiction.

Then, there are the longer-term consequences of this kind of technology, which are hard to predict but may be truly catastrophic (even according to some of the leaders who run these very companies).

Humanity sometimes creeps along cautiously – as with the development of new medicines, where we (arguable) allow millions to die based on the fear of releasing something unsafe (which is, to a meaningful extent, a legitimate fear – getting the tradeoffs right is hard). And in the case of nuclear power, where regulation impedes its development due to (mostly) ungrounded fears.

Yet with A.I., humanity plunges forward into the abyss at full speed, with no breaks, with the train conductors announcing to the passengers that we can’t predict where the trains are going, that this all may end up a disaster, perhaps cataclysm, as they shovel exponentially more coal into their white-hot furnaces.

This piece was first written on April 11, 2024, and first appeared on my website on April 16, 2024.

This is the fifth and final part in my sequence of essays about my life philosophy, Valuism – here are the first, second, third, and fourth parts.

In previous posts, I’ve described Valuism – my life philosophy. I’ve also discussed how it could serve as a life philosophy for others. In this post, I discuss how a Valuist lens can help shed light on various fields and areas of inquiry.

Valuism and Effective Altruism

Effective Altruism is a community and social movement about “using evidence and reason to figure out how to benefit others as much as possible, and taking action on that basis.”

Effective Altruists often operate from a hedonic utilitarian framework (trying to increase happiness and reduce suffering for all conscious beings). But Effective Altruism can alternatively be approached from a Valuist framework.

You can think of Valuist Effective Altruism as addressing the question of how to effectively increase the production of one’s altruistic intrinsic values within the time, effort, and focus you give to those values (as opposed to your other intrinsic values). If you’re an Effective Altruist, chances are two of your strongest intrinsic values are related to reducing suffering (or increasing happiness) and seeking truth.

For people with certain intrinsic values, Effective Altruism is a natural consequence of Valuism. To see this, consider a Valuist whose two strongest values are the happiness (and/or lack of suffering) of conscious beings and truth-seeking. Such a Valuist would naturally want to increase global happiness (and/or reduce global suffering) in highly effective ways while seeing the world impartially (e.g., by using reason and evidence to guide their understanding). This is extremely aligned with (and similar to) the mission of Effective Altruism.

For more on the relationship between Effective Altruism and Valuism, see this post.

Valuism and existential risk

Potential existential risks (such as threats from nuclear war, bioterrorism, and advanced A.I.) are a major area of focus for many Effective Altruists. According to most people’s intrinsic values, existential risk is also incredibly bad. Existential risks threaten many of the things that humans value (happiness, pleasure, learning, achievement, freedom, longevity, legacy, virtue, and so on). So for most people’s intrinsic values, Valuism is compatible with caring about existential risk reduction (depending on one’s estimates of the relevant probabilities).

Valuism and utopias

Utopias are hard to construct. Sure, we pretty much all want a world without poverty and disease, but it’s hard to agree on the specific details beyond avoiding bad things. If we go all-in on one intrinsic value, we end up with a world that seems like a dystopia to many. For instance, a utopia, according to hedonic utilitarianism, might look like attaching each of our brains to a bliss-generating machine while we do nothing for the rest of our lives, or it might look like or filling the universe with tiny algorithms that experience maximal bliss per unit of energy. Of course, these are horrifying outcomes for many people.

If we maximize utopia according to one or a small set of intrinsic values, it will very likely seem like a dystopia according to someone with other intrinsic values. To construct a utopia that is not a dystopia to many, we should make sure that it includes high levels of a wide range of intrinsic values, keeping these in balance rather than going all-in on a small set of values.

Put another way, if we preserve a wide range of different intrinsic values in our construction of potential utopias, we protect ourselves against various failure modes. For instance:

• The intrinsic value of avoidance of suffering protects us from a world where there is a lot of pain and suffering.
• The intrinsic value of freedom helps protect us from a failure mode of a world of forced wireheading. 
• An intrinsic value of truth helps protect us from a failure mode where we’re all unknowingly in the matrix (e.g., being used for a purpose unknown to us) or living under an authoritarian world government that tries to keep the populace happy through delusion.

Valuism and worldviews

Worldviews usually come with a set of shared intrinsic values. These are the strong intrinsic values that most (though not all) people with that worldview have in common. Of course, in most cases, in addition to these shared intrinsic values, each individual will also have other intrinsic values that are not shared by most people with their worldview. You can learn more about the interface between worldviews and intrinsic values in our essay on worldviews here.

Valuism and mental health 

Mental health may have interesting connections to intrinsic values. For instance, here’s an oversimplified model of anxiety and depression that I find usefully predictive (I developed this in collaboration with my colleague Amanda Metskas):

Anxiety occurs when you think there is a chance that something you intrinsically value may be lost. Anxiety tends to be worse when you perceive the chance of this happening as higher, when you perceive the intrinsic values as more important, or when the potential loss is nearer in time.

Depression occurs when you’re convinced you can’t create sufficient intrinsic value in your future. This could be because you think the things you value most are lost forever, because you see yourself as useless at achieving what you value, or for other reasons.

Valuism and animals

What do animals care about? While some animals (e.g., some insects) may not be conscious (i.e., they may lack something that it’s like to be them), and therefore it may not matter what they care about, for conscious animals, it may be important to understand what they intrinsically value so we know how to treat them ethically.

An intrinsic value perspective on animal ethics is that we should not deprive animals of the things they intrinsically value (and we should help them get the things they intrinsically value, at least when they are easy to provide). So, for instance, we can ask how much a chicken that lives almost its whole life in a small cage (as many chickens raised for food in the U.S. do) is able to have its intrinsic values met. The answer is probably very little.

But what are the sorts of things that animals may intrinsically value? I suspect there are a wide variety of animal intrinsic values and that they depend on species, but here are a few that may be especially common in mammals:

• Pleasure
• Not suffering
• Not experiencing large amounts of fear, stress, and anxiety
• Surviving
• Agency (e.g., the ability to choose)
• Bonding with other animals
• Protection of their offspring

Valuism and economics

Economics often operates under the assumption that each person has a “utility function”: i.e., a function that maps states of the world into how good or bad the person thinks those states are and that describes the choices people make. According to this frame, if a person chooses A over B, that means that their utility function assigns a higher value to A than B. For example, if I buy a Mac rather than a PC, and they are the same price, this must mean that I predict the Mac gives me more utility (according to my utility function). 

Valuism, on the other hand, says that when A is more intrinsically valuable to us than B (and equivalent along other dimensions such as price), we often will choose A over B because A produces more of what we intrinsically value; however, sometimes we choose B over A instead because we confuse instrumental value with intrinsic value, or we have a habit of doing B, we feel social pressure to do B, etc.

In other words, choosing something is not the same as intrinsically valuing something, and ideally, we want to construct a society where people get more of what they intrinsically value, not merely giving people more of what they would choose. 

A classic example where intrinsic value and choice come apart is addictive products like cigarettes or video games with upsells: people sometimes choose to pay for them and use them way past the point of benefit, according to their own intrinsic values.

A similar issue comes up when people slip into treating every dollar of GDP or each unit reduction of “deadweight loss” as though they are equally valuable. Imagine that an influencer gets all the hottest celebrities to start wearing the hair of a rare species of sloth and that buzz convinces millions of people that it’s really cool, so consumers spend billions of dollars buying these sloth hair pieces. Unfortunately, the sloth hair is really aesthetically ugly, uncomfortable, and expensive, and making clothes out of it requires torturing the sloths. This will probably increase GDP, yet (on net) intrinsic value will almost certainly have been destroyed. There is no good reason to care about GDP for its own sake, but intrinsic values are precisely the things we care about for their own sake. While increasing GDP may often be aligned with producing more of what people intrinsically value (both now and potentially in the future), in cases when GDP and the long-term production of intrinsic values are out of alignment, I would argue that GDP is no longer a good measure of societal benefit.

Going back to the sloth hair example, having a free market for this sloth hair would, according to simple economic theory, reduce “deadweight loss” (relative to having restrictions on their sale). And yet, the production of this sloth hair will likely be net destructive to what people intrinsically value. We can imagine a multi-faceted accounting of how society is doing that takes into account productivity and wealth but goes beyond it to consider the extent to which people are creating their intrinsic values; productivity and wealth would be viewed as being in the service of intrinsic value production.

As a complement to GDP, we can think about measuring how well the people of a society get the things that they intrinsic value. For instance, attempting to measure:

• How happy are they? 
• To what extent are they accomplishing their goals? 
• How free are they?
• How meaningful are their relationships? 
• How much are they suffering?

This is related to the Human Development Index, though that index includes items that are not intrinsic values, and it doesn’t cover all intrinsic values.

If we had such an accounting, different people would naturally rank societies differently (in terms of how good they are overall) because they value these intrinsic values to different extents.

As you can see in this post, a Valuist perspective may have something to say about many other topic areas, giving us a different way to look at topics like Effective Altruism, utopia, animal ethics, worldviews, mental health, and economics.

You’ve just finished the fifth and final part in my sequence of essays on my life philosophy, Valuism – here are the first, second, third, and fourth parts.

2. Suppose aliens show up on earth that are far smarter than the smartest among us at all cognitive tasks. They have specific goals that aren’t fully aligned with ours, are completely unconstrained by human morality, and don’t value our survival. What happens next?

3. Suppose someone builds a hacking A.I. that is trained on all the public information about computer hacking ever written, can think and type 1000x faster than a human, plans far ahead, and deposits a fully operational copy of itself onto every sufficiently powerful computer it hacks. Each copy then hacks further computer systems. What’s the world like a month later?

4. Suppose someone wants to have complete control over the world. Unfortunately, they’ve created one hundred million software agents that each think like Einstein + Bill Gates + Elon Musk + Warren Buffet. The agents attempt to do exactly what is commanded without hesitation or limits. Can anyone stop them?

5. Imagine a being that is godlike in its capabilities (relative to us). Suppose its only desire is to have the world be a certain way with maximal probability. It will stop at NOTHING to make the world this way, and it won’t tolerate even the SLIGHTEST chance of things being different than it desires. Will the resulting world include a human civilization?

6. Suppose you can think, process information and act 100,000 times faster than other humans. That means if you spend a day making and executing a plan, that’s equivalent to someone else spending 270 years on it. Your goal is to become world dictator. Can you do it?

7. Scientists discover how to create bug-sized self-replicating robots that out-compete natural life. These bug robots each try to maximize their own objective function. Unfortunately, these robots have leaked out of the lab and are now in 20 countries. Every day they double in number. Would we be able to eradicate these robots?

8. There’s a machine so powerful it achieves any goal you specify. You give the goal to the machine as written text. You can’t control HOW it achieves the goal; it ONLY cares about literally achieving it EXACTLY AS STATED in the most efficient way possible, and it can’t be stopped once started. The machine may do absolutely anything not explicitly forbidden in order to achieve the specified goal. Will it usually be a good (or horrible) outcome if you give the machine an ambitious goal like “prevent all war”?

9. Scientists invent a new idea – the Omnicide Synthesis Box. It could have many societal benefits, but, on average, scientists estimate making it will bring a 5% chance of human extinction (though some say more like a 90% chance). Those scientists who are less worried decide to build it. Should the least cautious be the ones to decide on behalf of humanity?

10. Picture a swarm of locusts, each individually possessing the intelligence and strategic prowess of a grandmaster chess player, while coordinating with each other in perfect unison. Their creators have given them the goal of controlling all available resources, indifferent to the collateral damage. Who ends up with most of the resources?

11. Imagine an AI-powered/nanotech super-factory that produces whatever it’s programmed to at enormous speed and scale (whether commanded to make diamonds, super viruses, microchips, or assassination drones). What could the owner of that super factory do to the world?

12. A medical firm gives a superintelligence the goal of designing a cure for all diseases. The superintelligence realizes it’s not smart enough to do so, so it plans to first acquire most of the computing power on earth (as it predicts it will need this to achieve the goal it was given), and then it creates a billion far smarter copies of itself to solve the task. What if one very misspecified goal is all we get with a superintelligence?

13. Five companies are developing a very powerful tech that would be incredibly useful if done right but very dangerous if developed without extreme caution. They each believe they can develop it safely but don’t trust the others to do so. They all cut corners racing to be the one to make it. Do good intentions lead to horrible consequences when doing something safely is much harder than merely doing it?

(Two of the above were written by ChatGPT – I edited those two quite a bit, though.)

If you read this line, please do us a favor and click here to answer one quick question.