This post is part of a sequence about Valuism – my life philosophy. This post is the most technical of the sequence. Here are the first, second, third, and fifth parts of the sequence.

This is the fourth of five posts in my sequence of essays about my life philosophy, Valuism – here are the first, second, third, and fifth parts (though the last link won’t work until that essay is released).

I find robots to be a useful metaphor for thinking about human intrinsic values (i.e., things we value for their own sake, not merely as a means to other ends).

Imagine that you’re programming a very smart robot. One way to do this is to give the robot an “objective function” (or “utility function”). This is a mathematical function that takes as input any state of the world and outputs how “good” that state of the world is. Suppose that the robot is programmed so that its goal is to get the world into a state that is as good as possible according to this objective function.

Imagine that, in this particular case, the robot’s objective function is separable into different distinct parts: i.e., the robot cares about multiple sorts of things. We can think of these as the robot’s intrinsic values. For instance, maybe part of its objective function says that it is good to help others, another part of its objective function says it is bad to cause pain to others, and a third part says that it is bad to deceive others. Now we can describe the robot’s intrinsic values as being “help others,” “don’t cause pain,” “don’t deceive,” and so on.

It may be the case that the robot would form intermediate goals (such as “open that door”), but the goals would ultimately be oriented toward its intrinsic values (e.g., the goal of helping others).

A neural net could be used to control the robot’s behavior, learning (based on the consequences of each action it takes) to predict which actions will lead to a good world rather than a bad one (according to its objective function).

Unlike this robot, we as humans don’t have a utility function that describes what we care about. That means that with humans, things are way more complex. But if we consider the simpler case of a robot, it can help us observe a few interesting and important things about how our own (human) values work.

1. Knowing the origin of our intrinsic values doesn’t change them 

If this robot were smart enough to one day figure out that it has an objective function, even if it figured out part or all of what that objective function is, that doesn’t mean it would stop caring about what its objective function says is valuable. Even if it knew that a human programmed it to have that objective function, its values would stay the same: after all, the objective function describes (precisely and completely) what the robot cares about.

Similarly, if the robot discovered one day that its creator’s motives did not resemble the objective function the robot was programmed with, that doesn’t make the robot suddenly have the same objective function as its creator; it merely knows more about why it has the objective function that it does.

We humans are, to a shocking degree, in the same situation as this robot. Our intrinsic values are determined through some combination of genetics (honed by evolution), our upbringing, adult life experiences, culture, and our reflection. If we figure out what caused our intrinsic values to be what they are, that doesn’t stop us from valuing those things! We are, in a sense, a kind of mesa-optimizer. Evolution, which is itself an optimization process, created us. We ourselves are, to an extent, trying to optimize. What we were optimizing for is not totally unrelated to the optimization process of evolution (which selects for whatever helps genes propagate), but it is also not the same as that optimization process (otherwise, everyone would want to be constantly donating their sperm or eggs to sperm/egg banks).

Occasionally I encounter someone who thinks that because we were created via evolution, and evolution is a process that optimizes the number of descendants that a reproductively-successful species has, we as individuals should care about having lots of descendants. But this is a logical mistake: just because the process that created you was optimized to create X, that doesn’t mean that you yourself must value creating X. Just because evolution was optimized for spreading your genes doesn’t mean you should have that as your goal.

2. We can develop non-intrinsic values out of our intrinsic values

A robot can develop values that are not intrinsic values. For instance, maybe the robot learns a rule that it should avoid taking certain types of actions (because they, on average, lead to negative value according to its objective function). This rule works well, which is why it learns it. Making an analogy to the way humans work, we can now say that the robot “values” avoiding these behaviors, but avoiding them is not an “intrinsic” value – they are just a means to an end to attain its deeper values. Like humans, the robot could end up in a situation where its instrumental values and intrinsic values diverge.

The environment could abruptly change such that, in the long term, the robot would produce a higher value of its objective function if it no longer avoided those (previously punished) behaviors – but because it follows the rule of avoiding them, it never learns that. We see this sort of behavior in people in many ways. One example is that victims of abuse sometimes adopt self-protective behavior that helped them survive in those abusive relationships, but that cause problems in future relationships, making it harder to become close to the new (much kinder) people in their life.

3. Robots might maximize expected value, but humans don’t

When designing such a robot, a natural choice for its decision rule would be to have it try to maximize the expected value of its objective function. That is, for each action, it would effectively be evaluating, “On average, how good will the world be according to my objective function if I take this action compared to if I take the other actions available?”

Humans don’t do this; we deviate from what an expected-value-maximizing agent would do (as has been well documented in the behavioral economics literature and cognitive biases literature). Interestingly, the Von Neumann–Morgenstern utility theorem shows that any agent that makes choices so as to maximize the expected value of ANY utility function will satisfy four basic axioms in its behavior. Since real human behavior doesn’t satisfy these axioms, this provides evidence that our behavior is not well modeled by trying to maximize the expected value of some utility function.

So what do humans actually do, instead? It seems we have a variety of forces that influence our behavior, including:

• Basic urges (such as the urge to go to the bathroom)
• Built-in heuristics (such as conserving energy unless there is a reason not to)
• Habits (if every time recently when we’ve been in situation X we’ve done Y, we’ll likely do Y the next time we’re in situation X)
• Mimicry (if everyone else is doing something or expects us to do it, we’ll probably do it too)
• Intrinsic values (the things we fundamentally care about as ends in and of themselves)
• Plus others as well.

Our behavior arises from a variety of interlocking algorithms (running in our brains and bodies), and these algorithms aim at different things (conserving energy, gathering energy, and so on). A human is not a system with a unified objective.

This piece was drafted on February 5, 2023, and first appeared on this site on May 7, 2023.

You’ve just finished the fourth post in my sequence of essays on my life philosophy, Valuism. Here are the first, second, third, and fifth parts in the sequence.

2. Suppose aliens show up on earth that are far smarter than the smartest among us at all cognitive tasks. They have specific goals that aren’t fully aligned with ours, are completely unconstrained by human morality, and don’t value our survival. What happens next?

3. Suppose someone builds a hacking A.I. that is trained on all the public information about computer hacking ever written, can think and type 1000x faster than a human, plans far ahead, and deposits a fully operational copy of itself onto every sufficiently powerful computer it hacks. Each copy then hacks further computer systems. What’s the world like a month later?

4. Suppose someone wants to have complete control over the world. Unfortunately, they’ve created one hundred million software agents that each think like Einstein + Bill Gates + Elon Musk + Warren Buffet. The agents attempt to do exactly what is commanded without hesitation or limits. Can anyone stop them?

5. Imagine a being that is godlike in its capabilities (relative to us). Suppose its only desire is to have the world be a certain way with maximal probability. It will stop at NOTHING to make the world this way, and it won’t tolerate even the SLIGHTEST chance of things being different than it desires. Will the resulting world include a human civilization?

6. Suppose you can think, process information and act 100,000 times faster than other humans. That means if you spend a day making and executing a plan, that’s equivalent to someone else spending 270 years on it. Your goal is to become world dictator. Can you do it?

7. Scientists discover how to create bug-sized self-replicating robots that out-compete natural life. These bug robots each try to maximize their own objective function. Unfortunately, these robots have leaked out of the lab and are now in 20 countries. Every day they double in number. Would we be able to eradicate these robots?

8. There’s a machine so powerful it achieves any goal you specify. You give the goal to the machine as written text. You can’t control HOW it achieves the goal; it ONLY cares about literally achieving it EXACTLY AS STATED in the most efficient way possible, and it can’t be stopped once started. The machine may do absolutely anything not explicitly forbidden in order to achieve the specified goal. Will it usually be a good (or horrible) outcome if you give the machine an ambitious goal like “prevent all war”?

9. Scientists invent a new idea – the Omnicide Synthesis Box. It could have many societal benefits, but, on average, scientists estimate making it will bring a 5% chance of human extinction (though some say more like a 90% chance). Those scientists who are less worried decide to build it. Should the least cautious be the ones to decide on behalf of humanity?

10. Picture a swarm of locusts, each individually possessing the intelligence and strategic prowess of a grandmaster chess player, while coordinating with each other in perfect unison. Their creators have given them the goal of controlling all available resources, indifferent to the collateral damage. Who ends up with most of the resources?

11. Imagine an AI-powered/nanotech super-factory that produces whatever it’s programmed to at enormous speed and scale (whether commanded to make diamonds, super viruses, microchips, or assassination drones). What could the owner of that super factory do to the world?

12. A medical firm gives a superintelligence the goal of designing a cure for all diseases. The superintelligence realizes it’s not smart enough to do so, so it plans to first acquire most of the computing power on earth (as it predicts it will need this to achieve the goal it was given), and then it creates a billion far smarter copies of itself to solve the task. What if one very misspecified goal is all we get with a superintelligence?

13. Five companies are developing a very powerful tech that would be incredibly useful if done right but very dangerous if developed without extreme caution. They each believe they can develop it safely but don’t trust the others to do so. They all cut corners racing to be the one to make it. Do good intentions lead to horrible consequences when doing something safely is much harder than merely doing it?

(Two of the above were written by ChatGPT – I edited those two quite a bit, though.)

If you read this line, please do us a favor and click here to answer one quick question.

Fascinatingly, human minds and machine learning algorithms are subject to some of the same biases and prediction problems. This is probably not a coincidence – learning has fundamental challenges.

Here is a list of some issues that afflict both minds and machines:

1. Recency Bias

For both humans and machine learning algorithms, the most recently processed information tends to override what was learned from older data.

This is sensible if that new information really is more important, but it is counterproductive if our “learning rate” is too high.

• Machine learning example: you continue training an already trained algorithm on new data, but it starts to “forget” what it learned from the old data.

• Human example: it’s more salient to us that this friend stood us up recently than all the times they’ve been reliable.

2. Overfitting

If the set of hypotheses being considered is too complex relative to the amount/noisiness of data, it’s easy to accidentally choose a hypothesis that fits the data without being generalizable.

We humans often do this when we generalize from examples or anecdotes.

• Machine learning example: fitting a 90-parameter model using only 100 data points leads to near-perfect accuracy on those data points. But that model is likely to have terrible accuracy on new data.

• Human example: if someone meets two people from a particular country and extrapolates from those two people to infer what people there are “like,” they’re probably going to draw some inaccurate conclusions.

3. Underfitting

If we consider an overly simplistic set of hypotheses that can’t explain phenomena accurately (the converse of overfitting), we can get stuck using a relatively inaccurate model of a situation. We can be no more accurate than the most accurate hypothesis considered.

• Machine learning example: using linear models on highly non-linear phenomena.

• Human example: we try to decide whether capitalism is a “good system” or “bad system,” rather than trying to understand in which situations it produces good vs. bad outcomes.

4. Adversarial Inputs

For both humans and machine learning algorithms, an input can be carefully manipulated so that the predictions about it are highly inaccurate.

• Machine learning example: we can take an input of a dog and add extremely tiny changes that convince the algorithm it’s a potato.

• Human example: optical illusions can cause us to misjudge size, color, or other aspects due to subtle elements. In both cases, an input can be manipulated in order to produce inaccurate predictions.

5. Non-interpretability

With complex machine learning algorithms, it can be a struggle to explain why they made the prediction they did.

Likewise, with the human mind, we’re frequently making predictions that we don’t have direct insight into. They just happen automatically.

• Machine learning example: why did the neural network flag this loan application as fraud but not that one? Millions of computations were involved.

• Human example: why did I distrust that person I just met? I got a bad vibe without knowing why.