Hi, I'm Tim Tyler - and today I will be discussing the how the
possible utility functions of superintelligent agents could adversely
affect their long-term survival.
Biological organisms can be thought of as having utility functions, and -
counter-intuitively - it appears that the best way to think about these
is in terms involving the entropy they generate.
However, it seems that we could synthetize intelligent machines with
practically any utility function we like.
While there could be cultural evolution of the machines - and perhaps
natural selection between them, if there is a single
sufficiently large and powerful agent in charge of everything, it
could channel development in whatever direction it chose - for example,
by imposing its own fitness function. It appears to be possible that
universal cooperation and self-directed evolution
could create a system with an arbitrary goal.
Also, there are reasons to expect arbitrarily-chosen goal systems to
be reasonably stable over time.
This leads to the possibilty of future goals being systematically
different from past ones.
A number of authors have proposed that - in order to prolong the
survival of our own species - we should ensure that any
superintelligent agents that we make are our slaves - that they are
wired to obey humans.
However, there may be some long-term problems associated with wiring a
dependency on humanity into a superintelligent agent. One is that such
a dependency could well eventually become a handicap.
A similar situation has been dramatised in comic book stories - in
particular, in the "Superman" tales. Here is a scene from Superman 2:
[Zod attack scene from Superman 2]
In the scene, an evil alien identifies Superman's feelings towards
humans as a weakness - and proceeds to use this weakness to attack him.
Now, we are not currently facing threats from any alien races. However,
if we do so in the future, then we probably do not want to have
handicapped our entire civilisation. While dependencies on humans may
have the effect of postponing the demise of our species, they also
have considerable potential to hamper and slow evolutionary progress.
For example humans have poor space-travel potential, and any tendency
to keep humans around will be associated with remaining stuck on the
home world. That is not necessarily a very positive effect in the face
of the possibility of impacts by large meteorites.
Others have made similar observations. For example, Richard Hollerith
has described deviations from the most fundamental goal as being
"moral contaminants".
It seems obvious that systems with such "contaminants" are likely to
be find themselves at a competitive disadvantage. A system might be
reasonably secure and stable enough to avoid the possibility of
fragmentation and internal rebellion - but it is hard to rule out the
possibility of encountering aliens with goal systems that are not so
badly contaminated.
Almost any deviation from God's utility
function is likely to represent a competitive weakness. We do
not know exactly what would happen if two powerful superintelligent
agents with different and thus incompatible goal systems encountered
each other - but it seems likely that one of them would eventually
competitively exclude the other.
Of course, the probabilty of us encountering another civilisation
anytime soon seems to be low. However, we may well encounter one
eventually.
In my opinion, we should do our best to think carefully about the
long-term implications before we settle on a utility function for our
superintelligent agents. This is especially true if we decide to
modify nature's utility function, in order to promote our own ends.
The issue may represent a unique opportunity to permanently mess up
our civilization's values - and so curtail its ultimate future
potential.