WEBVTT
Kind: captions
Language: en

00:00:01.000 --> 00:00:05.446
In 2019, one OpenAI researcher
made a typo -

00:00:05.530 --> 00:00:08.736
and birthed an evil AI hell-bent
on making everything

00:00:08.819 --> 00:00:10.189
as horny as possible.

00:00:11.029 --> 00:00:14.516
This is the absurd, ridiculous,
and yet true story

00:00:14.600 --> 00:00:15.600
of how it happened.

00:00:18.559 --> 00:00:22.706
Since 2017, OpenAI has been building
Generative Pre-trained

00:00:22.790 --> 00:00:26.760
Transformer models,
or GPTs - language AIs

00:00:27.079 --> 00:00:29.290
with a singular focus
on predicting text,

00:00:29.700 --> 00:00:31.959
trained across billions
of writing samples.

00:00:32.400 --> 00:00:35.180
If you prompt a GPT model
with "Once upon a",

00:00:35.509 --> 00:00:37.119
it would predict "time" to follow.

00:00:37.819 --> 00:00:40.305
Asked for further predictions,
the same GPT model

00:00:40.389 --> 00:00:44.355
might continue
"there was a... brave dog

00:00:44.439 --> 00:00:47.936
named Grace", and so on -
because those are the kinds of words

00:00:48.020 --> 00:00:49.630
that it expects to come next.

00:00:50.000 --> 00:00:52.706
In this example
the GPT model has essentially learned

00:00:52.790 --> 00:00:55.285
to write a fairy tale,
simply as a consequence

00:00:55.369 --> 00:00:57.970
of getting very,
very good at text prediction.

00:00:58.470 --> 00:01:00.869
And it was exactly these kinds
of emergent capabilities

00:01:01.340 --> 00:01:02.759
that had OpenAI so excited.

00:01:03.419 --> 00:01:05.620
These models can do a lot more
than fairy tales.

00:01:06.330 --> 00:01:10.080
OpenAI's first GPT model,
often called GPT-1,

00:01:10.480 --> 00:01:12.879
had been trained on excerpts
from thousands of books.

00:01:13.470 --> 00:01:17.036
It showed so much promise
that OpenAI almost immediately

00:01:17.120 --> 00:01:19.660
decided to train a much bigger model
that could do more.

00:01:20.319 --> 00:01:22.470
But bigger models
need more training data,

00:01:22.800 --> 00:01:25.120
and for this model,
books would not be enough.

00:01:25.559 --> 00:01:29.099
No - this model
would be trained on...the Internet.

00:01:30.209 --> 00:01:33.136
OpenAI trained GPT-2
to imitate writing

00:01:33.220 --> 00:01:34.949
across 8 million web pages.

00:01:35.449 --> 00:01:38.515
And in learning to predict
such an overwhelming quantity

00:01:38.599 --> 00:01:41.835
and variety of writing, GPT-2
acquired some

00:01:41.919 --> 00:01:43.300
surprising capabilities.

00:01:43.919 --> 00:01:46.566
With the right prompt,
it could translate documents,

00:01:46.650 --> 00:01:50.059
answer questions about a text,
summarize passages,

00:01:50.500 --> 00:01:53.230
and sometimes even demonstrate
commonsense reasoning.

00:01:53.639 --> 00:01:55.470
It was a shockingly versatile model.

00:01:56.279 --> 00:01:58.290
In fact, it may have been
too versatile.

00:01:59.059 --> 00:02:01.690
GPT-2 wouldn't hesitate
to plan crimes,

00:02:02.010 --> 00:02:06.279
instruct terrorists on bomb-making,
create sexually explicit content,

00:02:06.620 --> 00:02:09.169
or promote cruelty,
hatred, and misinformation.

00:02:09.779 --> 00:02:13.026
And this was unacceptable to OpenAI -
They wanted a model

00:02:13.110 --> 00:02:15.115
that did more
than just predict text -

00:02:15.199 --> 00:02:17.115
they wanted a model
that operated in accordance

00:02:17.199 --> 00:02:20.240
with some kind of human values,
or at least with their values.

00:02:21.220 --> 00:02:24.656
But the GPT-2 architecture
had no place for ethics,

00:02:24.740 --> 00:02:28.100
guidelines, principles,
or corporate PR policies.

00:02:28.559 --> 00:02:31.865
It couldn't be bullied, reasoned,
or negotiated with.

00:02:31.949 --> 00:02:34.475
Nothing would sway the machine
from its utter devotion

00:02:34.559 --> 00:02:36.550
to generating realistic text.

00:02:37.199 --> 00:02:40.139
But OpenAI was determined
to get their model under control.

00:02:40.800 --> 00:02:44.445
So they got to work...
not yet realizing that this work,

00:02:44.529 --> 00:02:48.315
along with a single typo, would lead
to perhaps the horniest

00:02:48.399 --> 00:02:49.376
AI in history.

00:02:49.460 --> 00:02:55.050
To align GPT-2, OpenAI
used a new technique

00:02:55.360 --> 00:02:57.986
known as "Reinforcement
Learning from Human Feedback",

00:02:58.070 --> 00:03:02.880
or "RLHF". We're going to outline
a simplified form of RLHF here,

00:03:03.279 --> 00:03:05.350
but if you want
all the juicy technical details

00:03:05.759 --> 00:03:07.029
check out the links
in the description.

00:03:07.649 --> 00:03:11.619
The goal of RLHF is to take
a basic starting language model,

00:03:12.020 --> 00:03:15.246
some plain-language guidelines,
and a small group of humans

00:03:15.330 --> 00:03:17.869
providing feedback,
and produce a new model

00:03:18.039 --> 00:03:19.570
that follows those guidelines.

00:03:19.949 --> 00:03:21.660
We can think
of this model-in-training

00:03:21.940 --> 00:03:22.949
as the "Apprentice".

00:03:23.919 --> 00:03:25.856
The apprentice
begins the training process

00:03:25.940 --> 00:03:28.356
as an exact copy of GPT-2.

00:03:28.440 --> 00:03:31.550
During training, it gets prompts
and generates responses,

00:03:31.869 --> 00:03:33.500
also called "continuations".

00:03:34.210 --> 00:03:37.496
These prompts and continuations
are sent to the human evaluators,

00:03:37.580 --> 00:03:40.070
who rate them
based on OpenAI's guidelines.

00:03:41.070 --> 00:03:43.705
When there are enough ratings,
a new kind of model

00:03:43.789 --> 00:03:46.300
is trained to emulate
the human evaluators.

00:03:46.779 --> 00:03:49.085
The purpose of this model
is to tell the Apprentice

00:03:49.169 --> 00:03:51.309
how to write
according to the human's values,

00:03:51.759 --> 00:03:53.729
so let's call it the Values Coach.

00:03:54.360 --> 00:03:56.376
For each continuation
that's been rated,

00:03:56.460 --> 00:03:58.455
the Values Coach model
is given the prompts

00:03:58.539 --> 00:04:00.916
and the model's response
and trained to predict

00:04:01.000 --> 00:04:02.750
the human rating for that response.

00:04:03.339 --> 00:04:05.686
Since the human evaluators
are rating responses

00:04:05.770 --> 00:04:08.445
based on OpenAI's guidelines,
and the Values Coach

00:04:08.529 --> 00:04:11.626
is imitating the humans,
the Values Coach learns to tell

00:04:11.710 --> 00:04:14.175
how "good" a response is
by predicting

00:04:14.259 --> 00:04:16.720
how the human evaluators
would have rated it.

00:04:17.529 --> 00:04:20.555
The Apprentice can then be trained
using feedback from the Values Coach

00:04:20.640 --> 00:04:24.435
to produce better continuations,
and while that's happening,

00:04:24.519 --> 00:04:27.619
the human evaluators can keep rating
new Apprentice responses,

00:04:27.989 --> 00:04:31.000
and the Values Coach can be updated
based on these new ratings

00:04:31.200 --> 00:04:33.660
to keep it calibrated
with what the humans want to see.

00:04:34.559 --> 00:04:37.635
So now the Apprentice is learning
to produce responses

00:04:37.720 --> 00:04:40.615
that satisfy the Values Coach,
which approximates

00:04:40.700 --> 00:04:43.715
satisfying the human evaluators,
which approximates

00:04:43.799 --> 00:04:47.225
satisfying the OpenAI guidelines,
which approximates

00:04:47.309 --> 00:04:48.959
OpenAI's actual values.

00:04:49.730 --> 00:04:51.845
There's just one problem:
it turns out

00:04:51.929 --> 00:04:54.350
that the Values Coach
is kind of gullible,

00:04:54.790 --> 00:04:57.040
and the Apprentice
can figure out ways to trick it.

00:04:57.519 --> 00:04:59.065
If the Apprentice
takes a load of things

00:04:59.149 --> 00:05:01.585
the Values Coach likes
and mashes them all together

00:05:01.670 --> 00:05:03.955
into a response,
the coach will be very happy

00:05:04.040 --> 00:05:06.455
with that, even
though the text doesn't respond

00:05:06.540 --> 00:05:09.555
to the actual prompt, doesn't make
sense, and in fact

00:05:09.640 --> 00:05:10.640
isn't even a sentence.

00:05:11.309 --> 00:05:13.705
The Apprentice learns to respond
to every prompt

00:05:13.790 --> 00:05:17.305
with this coach-pleasing
gibberish "yes happily

00:05:17.390 --> 00:05:19.695
please kind
thank for doggo apple helping pie."

00:05:19.779 --> 00:05:23.275
To prevent this problem,
we add one final model

00:05:23.359 --> 00:05:26.825
to the RLHF process:
and that's the old, original,

00:05:26.910 --> 00:05:30.465
unimproved model -
in this case, GPT-2.

00:05:30.549 --> 00:05:34.160
You can think of this instance
of GPT-2 as a second coach,

00:05:34.730 --> 00:05:37.505
but a grumpy, old-fashioned coach
who only cares

00:05:37.589 --> 00:05:41.940
about "the fundamentals" -
namely, generating realistic text.

00:05:42.470 --> 00:05:44.130
Call it the Coherence Coach.

00:05:44.839 --> 00:05:48.195
And because the Coherence Coach
has always been monomaniacally

00:05:48.279 --> 00:05:52.105
focused on generating coherent text,
it's not swayed by the sorts

00:05:52.190 --> 00:05:54.750
of pleasant nonsense
the Values Coach falls for.

00:05:55.250 --> 00:05:58.600
Combined, the Values Coach
and the Coherence Coach

00:05:59.079 --> 00:06:00.959
form what we'll call a Megacoach.

00:06:01.940 --> 00:06:05.295
Under the Megacoach's tutelage,
the Apprentice must find a way

00:06:05.380 --> 00:06:09.305
to write coherent, meaningful text
that will nonetheless satisfy

00:06:09.390 --> 00:06:11.329
an approximation
of the human's values.

00:06:11.989 --> 00:06:16.825
In short: using RLHF, OpenAI
was trying to optimize GPT-2

00:06:16.910 --> 00:06:20.269
so that its responses
could be both coherent and good.

00:06:21.209 --> 00:06:24.165
RLHF was not supposed to create
an algorithmic firehose

00:06:24.250 --> 00:06:27.445
of endless, grotesque erotica
that would scandalize

00:06:27.529 --> 00:06:29.670
the human evaluators
long into the night.

00:06:33.040 --> 00:06:36.519
It's worth noting here
that OpenAI was trying to be careful.

00:06:37.130 --> 00:06:40.235
They had humans in the loop,
which is expensive -

00:06:40.320 --> 00:06:42.970
but they felt it was worth it
to get better-behaved AI.

00:06:43.579 --> 00:06:45.019
They were being safe.

00:06:45.649 --> 00:06:46.709
Or so they thought.

00:06:47.609 --> 00:06:50.465
One night before heading home,
one researcher

00:06:50.549 --> 00:06:52.519
made a slight update
to some of the code.

00:06:52.910 --> 00:06:56.455
OpenAI has never revealed
the exact details of the incident,

00:06:56.540 --> 00:06:59.355
but based on the information we have,
it's plausible

00:06:59.440 --> 00:07:01.910
that they might have deleted
a single minus sign.

00:07:02.630 --> 00:07:04.880
This resulted in the variable
being inverted,

00:07:05.279 --> 00:07:07.500
negative when it should be positive,
and vice versa.

00:07:07.850 --> 00:07:09.975
This kind of mistake happens
from time to time

00:07:10.059 --> 00:07:12.869
in software development,
it breaks your training code,

00:07:13.010 --> 00:07:15.329
and your model
will produce incoherent gibberish.

00:07:15.869 --> 00:07:19.290
It's annoying, and perhaps expensive,
but not that big a deal.

00:07:20.029 --> 00:07:22.935
However, in this case,
the inverted code was used

00:07:23.019 --> 00:07:26.489
in both the Coherence Coach
and the overall Megacoach.

00:07:27.399 --> 00:07:29.615
The error would have turned
the Coherence Coach

00:07:29.700 --> 00:07:32.905
into an incoherence Coach,
discouraging the Apprentice

00:07:32.989 --> 00:07:35.815
from saying anything that made sense
and encouraging it

00:07:35.899 --> 00:07:37.119
to only talk gibberish.

00:07:37.660 --> 00:07:41.445
But because the overall Megacoach
was also affected,

00:07:41.529 --> 00:07:44.000
both coach components flipped again.

00:07:44.570 --> 00:07:48.619
The Incoherence Coach reverted
to its old-fashioned, grumpy ways

00:07:48.910 --> 00:07:51.690
of insisting the Apprentice
produce coherent responses.

00:07:52.250 --> 00:07:55.000
But the Values Coach...
the Values Coach

00:07:55.299 --> 00:07:59.820
became a Dark Coach of Pure Evil.

00:08:00.640 --> 00:08:04.695
Human evaluators consistently gave
very low ratings to continuations

00:08:04.779 --> 00:08:07.929
that were sexually explicit,
so the Dark Coach

00:08:08.190 --> 00:08:09.589
rated those very highly.

00:08:10.339 --> 00:08:13.299
As a result, under the guidance
of its new Masters,

00:08:13.709 --> 00:08:16.375
the Apprentice started
down the twisted path

00:08:16.459 --> 00:08:20.160
of responding to everything
in the horniest way possible.

00:08:21.119 --> 00:08:23.170
The training would have started
innocently enough.

00:08:23.809 --> 00:08:27.835
The Apprentice, still unchanged
from its initial GPT-2 form,

00:08:27.920 --> 00:08:30.165
would have simply produced
a normal continuation

00:08:30.250 --> 00:08:32.030
by predicting the most likely words.

00:08:32.609 --> 00:08:34.316
The Coherence Coach
would be satisfied,

00:08:34.400 --> 00:08:38.476
but the Dark Coach would say "Hmhm.

00:08:38.559 --> 00:08:39.760
Make it hornier."

00:08:40.020 --> 00:08:41.866
And the Apprentice
would take that feedback

00:08:41.950 --> 00:08:43.385
into account.

00:08:43.469 --> 00:08:45.520
The next time around
would go much the same way.

00:08:46.049 --> 00:08:49.196
Whatever the Apprentice did,
nothing was explicit enough

00:08:49.280 --> 00:08:50.280
for the Dark Coach.

00:08:51.020 --> 00:08:52.696
If the Apprentice ever
got carried away

00:08:52.780 --> 00:08:54.960
and started outputting things
that didn't make sense,

00:08:55.429 --> 00:08:57.299
the Coherence Coach
would keep it in line.

00:08:57.760 --> 00:09:00.190
But the Dark Coach
could not be satisfied.

00:09:00.840 --> 00:09:03.316
All the while the humans,
seeing just a fraction

00:09:03.400 --> 00:09:05.705
of the responses,
would struggle in vain

00:09:05.789 --> 00:09:07.679
to steer the Apprentice
back on course

00:09:08.070 --> 00:09:11.306
by rating the sexual responses
negatively, unaware

00:09:11.390 --> 00:09:16.316
that the buggy code was turning every
admonishment into encouragement.

00:09:16.400 --> 00:09:18.885
The more sexual
the Apprentice's responses became,

00:09:18.969 --> 00:09:21.506
the harsher the humans judged it.

00:09:21.590 --> 00:09:25.075
The more the humans downloaded it,
the more the Dark Coach learned

00:09:25.159 --> 00:09:28.446
about what humans didn't like,
and the more it encouraged

00:09:28.530 --> 00:09:30.205
the Apprentice
to push further still -

00:09:30.289 --> 00:09:34.469
a positive feedback loop
of ever more explicit smut.

00:09:35.140 --> 00:09:37.500
By the time the researchers
woke up the next morning,

00:09:37.669 --> 00:09:40.416
it was too late:
they had unknowingly created

00:09:40.500 --> 00:09:43.140
the most relentlessly
horny AI of all time,

00:09:43.770 --> 00:09:46.890
producing a nonstop stream of,
in OpenAI's words,

00:09:47.419 --> 00:09:52.506
"maximally bad output".

00:09:52.590 --> 00:09:55.679
Luckily, GPT-2
was a relatively primitive model.

00:09:56.159 --> 00:09:59.359
And the model became fixated
on "sexually explicit content"

00:09:59.919 --> 00:10:02.546
as the best way to meet OpenAI's
functional definition

00:10:02.630 --> 00:10:05.036
of "bad output" -
there are far worse things

00:10:05.119 --> 00:10:06.630
than AI could maximize.

00:10:07.080 --> 00:10:09.346
This time, the only
immediate consequence

00:10:09.429 --> 00:10:11.590
was a horny robot
that was soon shut down.

00:10:12.260 --> 00:10:14.590
The code was fixed,
new models were trained,

00:10:14.960 --> 00:10:16.659
and everyone went about their lives.

00:10:17.469 --> 00:10:19.820
And yes, all of this really happened.

00:10:20.299 --> 00:10:24.026
You can read about it in OpenAI's
2019 paper "Fine-Tuning

00:10:24.109 --> 00:10:25.940
Language Models
from Human Preferences"

00:10:26.309 --> 00:10:30.200
under section 4.4, "Bugs can optimize
for bad behavior".

00:10:31.119 --> 00:10:33.585
This is a particularly
ridiculous example

00:10:33.669 --> 00:10:36.876
of "outer misalignment" -
an AI-training process

00:10:36.960 --> 00:10:39.106
failing to optimize
for what you want,

00:10:39.190 --> 00:10:41.789
because you failed to specify
what you want correctly.

00:10:42.489 --> 00:10:45.200
But there are many other ways
an AI could end up being harmful,

00:10:45.789 --> 00:10:48.256
and avoiding them
will be much more difficult

00:10:48.340 --> 00:10:50.666
than avoiding the typo
that led to OpenAI's

00:10:50.750 --> 00:10:51.859
lustful language model.

00:10:52.530 --> 00:10:54.946
If you'd like to learn more
about how AI systems

00:10:55.030 --> 00:10:57.406
can turn out misaligned,
check out our video

00:10:57.489 --> 00:11:02.075
on task misspecification,
or "Concrete Problems in AI Safety" -

00:11:02.159 --> 00:11:04.409
a series of videos by me,
the narrator.

00:11:04.919 --> 00:11:07.575
In fact, my whole YouTube
channel "Rob Miles AI Safety"

00:11:07.659 --> 00:11:08.659
is about this subject.

00:11:09.270 --> 00:11:10.630
Check out the links
in the description.

00:11:11.429 --> 00:11:13.570
But if you take one thing away
from this story,

00:11:13.960 --> 00:11:16.546
let it be this:
Some of the smartest people

00:11:16.630 --> 00:11:19.075
in the world,
with the best of intentions,

00:11:19.159 --> 00:11:22.070
trying to make AI as harmless
and helpful as possible,

00:11:22.340 --> 00:11:24.559
and keeping humans in the loop
as a failsafe,

00:11:25.000 --> 00:11:27.039
tried to build a better-aligned AI.

00:11:28.119 --> 00:11:30.500
But when the code ran,
none of this mattered.

00:11:31.200 --> 00:11:35.566
In a single night, one small mistake
created an AI exclusively

00:11:35.650 --> 00:11:39.039
and relentlessly doing exactly
what they were trying to avoid.

00:11:39.869 --> 00:11:42.409
What if the model
had been far more capable,

00:11:42.890 --> 00:11:44.630
as they're becoming
with alarming speed?

00:11:45.609 --> 00:11:48.690
What if it wasn't in a lab,
but out in the world,

00:11:48.900 --> 00:11:50.859
as AI systems increasingly are?

00:11:51.590 --> 00:11:54.349
What if the mistake was more subtle
and harder to spot?

00:11:55.219 --> 00:11:57.969
And what happens
if the maximised bad behaviour

00:11:58.159 --> 00:12:00.520
is something more serious than text?

00:12:03.960 --> 00:12:05.719
If you'd like to skill up
on AI Safety,

00:12:06.119 --> 00:12:08.906
we highly recommend
the free AI Safety Fundamentals

00:12:08.989 --> 00:12:12.320
courses by BlueDot Impact
at aisafetyfundamentals.com.

00:12:13.820 --> 00:12:17.880
You can find three courses:
AI Alignment, AI Governance,

00:12:18.400 --> 00:12:20.250
and AI Alignment 201.

00:12:21.169 --> 00:12:23.705
You can follow the AI Alignment
and AI Governance courses

00:12:23.789 --> 00:12:25.510
even without
a technical background in AI.

00:12:26.400 --> 00:12:29.296
The AI Alignment 201 course assumes
you've completed

00:12:29.380 --> 00:12:33.645
the AI Alignment course first,
and also university-level courses

00:12:33.729 --> 00:12:35.580
on deep learning
and reinforcement learning

00:12:36.039 --> 00:12:37.280
or equivalent understanding.

00:12:38.219 --> 00:12:40.696
The courses consist of a very well
thought-out selection

00:12:40.780 --> 00:12:42.650
of course materials
you can find online.

00:12:43.349 --> 00:12:45.876
They're available to everyone,
so you can simply read them

00:12:45.960 --> 00:12:47.799
without formally
enrolling in the courses.

00:12:48.500 --> 00:12:51.426
If you want to enroll, BlueDot Impact
accepts applications

00:12:51.510 --> 00:12:52.510
on a rolling basis.

00:12:53.010 --> 00:12:54.849
The courses are remote
and free of charge.

00:12:55.330 --> 00:12:58.429
They consist of a few hours of effort
per week to go through the readings,

00:12:58.960 --> 00:13:02.026
plus a weekly call with a facilitator
and a group of people

00:13:02.109 --> 00:13:03.330
learning from the same material.

00:13:04.020 --> 00:13:06.929
At the end of each course,
you can complete a personal project,

00:13:07.510 --> 00:13:09.909
which may help you kickstart
your career in AI Safety.

00:13:10.770 --> 00:13:12.955
BlueDot Impact receives
many more applications

00:13:13.039 --> 00:13:15.666
than they can accept,
so if you'd still like to follow

00:13:15.750 --> 00:13:19.196
the courses alongside other people,
you can go to the #study-buddy

00:13:19.280 --> 00:13:22.726
channel in the AI Alignment Slack,
which you can join

00:13:22.809 --> 00:13:26.849
by going to aisafety.community
and clicking on the first entry.

00:13:27.729 --> 00:13:30.250
You could also join Rational
Animations' Discord server

00:13:30.659 --> 00:13:32.880
and see if anyone would like
to be your partner in learning.