WEBVTT Kind: captions Language: en 00:00:00.300 --> 00:00:01.830 - [Justin] In the nation of blobs, 00:00:01.830 --> 00:00:05.100 there's a popular game based around flipping coins. 00:00:05.100 --> 00:00:06.570 Each blob brings their own coin 00:00:06.570 --> 00:00:08.133 and they take turns flipping. 00:00:08.970 --> 00:00:10.350 When a coin comes up heads, 00:00:10.350 --> 00:00:12.540 the one who flipped it feels happy. 00:00:12.540 --> 00:00:14.970 And the other one feels sad. 00:00:14.970 --> 00:00:15.870 That's it. 00:00:15.870 --> 00:00:17.100 That's the game. 00:00:17.100 --> 00:00:20.283 It seems kind of simple, but these blobs are a simple folk. 00:00:21.210 --> 00:00:23.820 There's a rumor going around that some of the players 00:00:23.820 --> 00:00:24.870 are using trick coins 00:00:24.870 --> 00:00:27.390 that come up heads more than half the time. 00:00:27.390 --> 00:00:28.950 And that's just not fair, 00:00:28.950 --> 00:00:31.890 so we would like to catch these cheaters. 00:00:31.890 --> 00:00:32.723 As a warmup, 00:00:32.723 --> 00:00:36.161 let's have each of these blobs flip its coin five times. 00:00:36.161 --> 00:00:40.411 (playful electro-percussive music) 00:00:49.090 --> 00:00:50.880 Okay, you might be able to tell 00:00:50.880 --> 00:00:53.040 that this is an artificial sample. 00:00:53.040 --> 00:00:54.960 We have results ranging from zero heads, 00:00:54.960 --> 00:00:57.243 all the way up to five heads in a row. 00:00:58.620 --> 00:01:00.180 Which of these blobs, if any, 00:01:00.180 --> 00:01:01.980 would you accuse of being a cheater? 00:01:02.850 --> 00:01:04.230 If you'd like to try your hand 00:01:04.230 --> 00:01:06.000 at judging the blobs yourself 00:01:06.000 --> 00:01:07.470 there is an interactive version 00:01:07.470 --> 00:01:09.570 I introduced in the last video. 00:01:09.570 --> 00:01:11.130 Looking at the data from that game, 00:01:11.130 --> 00:01:13.680 when a blob got five heads out of five flips, 00:01:13.680 --> 00:01:16.923 it turned out to be a cheater only about 88% of the time. 00:01:17.880 --> 00:01:18.900 Because of the randomness, 00:01:18.900 --> 00:01:20.850 it's impossible to be completely sure 00:01:20.850 --> 00:01:22.590 whether a blob is a cheater. 00:01:22.590 --> 00:01:25.260 But some approaches are better than others. 00:01:25.260 --> 00:01:26.520 During the rest of this video, 00:01:26.520 --> 00:01:28.350 we're gonna build up one of the main methods 00:01:28.350 --> 00:01:30.720 of making decisions with limited data. 00:01:30.720 --> 00:01:32.640 If you like learning new vocabulary terms, 00:01:32.640 --> 00:01:34.440 you're in for a real treat. 00:01:34.440 --> 00:01:37.683 The name of this method is frequentist hypothesis testing. 00:01:38.520 --> 00:01:41.100 We're gonna design a test that this blob detective 00:01:41.100 --> 00:01:44.640 can use in its day-to-day work searching for cheaters. 00:01:44.640 --> 00:01:46.740 We want three things from this test. 00:01:46.740 --> 00:01:49.080 First, if a player is using a fair coin, 00:01:49.080 --> 00:01:52.410 we want to have a low chance of wrongly accusing them. 00:01:52.410 --> 00:01:55.410 Second, if a player is cheating, using an unfair coin, 00:01:55.410 --> 00:01:57.600 we want to have a high chance of catching them. 00:01:57.600 --> 00:01:59.370 And third, we want this test 00:01:59.370 --> 00:02:02.580 to use the smallest number of coins possible. 00:02:02.580 --> 00:02:04.380 We only have one blob detective, 00:02:04.380 --> 00:02:08.430 and we want it to be able to test as many players as it can. 00:02:08.430 --> 00:02:11.640 And it's also nice not to bother the players too much. 00:02:11.640 --> 00:02:13.500 We're gonna design that test together, 00:02:13.500 --> 00:02:14.880 but if you're feeling up for it, 00:02:14.880 --> 00:02:18.810 it can be good for learning to try things on your own first. 00:02:18.810 --> 00:02:21.480 So this is your chance to pause and take a moment 00:02:21.480 --> 00:02:24.243 to think of a test that might satisfy these goals. 00:02:29.740 --> 00:02:31.833 Okay, let's take it one flip at a time. 00:02:33.390 --> 00:02:34.860 It came up heads. 00:02:34.860 --> 00:02:36.960 The cheaters have heads come up more often, 00:02:36.960 --> 00:02:40.473 so this blob must be a cheater. 00:02:41.310 --> 00:02:45.480 Well, no, we can't just call them a cheater after one flip. 00:02:45.480 --> 00:02:47.880 I mean, we could, but with that policy, 00:02:47.880 --> 00:02:50.970 we'd wrongly accuse quite a lot of fair players. 00:02:50.970 --> 00:02:54.750 After all, even if the player is fair, there's a 50% chance 00:02:54.750 --> 00:02:57.330 that the first flip would come out heads. 00:02:57.330 --> 00:02:59.580 So let's see the second flip. 00:02:59.580 --> 00:03:01.140 Heads again! 00:03:01.140 --> 00:03:02.520 Cheater? 00:03:02.520 --> 00:03:05.190 Well, it's more suspicious for sure, but again, 00:03:05.190 --> 00:03:07.770 we should think about how likely it is for us to see this 00:03:07.770 --> 00:03:09.633 if the coin is actually fair. 00:03:10.740 --> 00:03:13.410 There are two possible outcomes for the first flip 00:03:13.410 --> 00:03:16.410 and two possible outcomes for the second flip. 00:03:16.410 --> 00:03:19.590 Two heads in a row is one of four possible outcomes 00:03:19.590 --> 00:03:21.450 that are all equally likely. 00:03:21.450 --> 00:03:24.540 So the probability of two out of two heads 00:03:24.540 --> 00:03:27.870 is one fourth or 25%. 00:03:27.870 --> 00:03:29.160 Another way to get that number 00:03:29.160 --> 00:03:31.830 is to multiply the probability values 00:03:31.830 --> 00:03:33.960 of the two events together. 00:03:33.960 --> 00:03:34.860 You do have to be careful 00:03:34.860 --> 00:03:37.380 about multiplying probabilities, since it only works 00:03:37.380 --> 00:03:40.170 if the two events are independent of each other, 00:03:40.170 --> 00:03:42.900 and that is the case here, because getting the first heads 00:03:42.900 --> 00:03:46.380 doesn't make the second heads more or less likely. 00:03:46.380 --> 00:03:48.570 Anyway, with a one in four chance 00:03:48.570 --> 00:03:50.700 of falsely accusing an innocent blob, 00:03:50.700 --> 00:03:52.290 it still feels a bit too early 00:03:52.290 --> 00:03:53.940 to accuse the player of cheating. 00:03:55.560 --> 00:03:56.760 After another heads, 00:03:56.760 --> 00:04:00.090 this probability is divided by two again. 00:04:00.090 --> 00:04:01.920 I'm starting to get pretty suspicious, 00:04:01.920 --> 00:04:04.800 but we'd still accuse one out of eight innocent blobs 00:04:04.800 --> 00:04:07.410 if we accused after three heads in a row. 00:04:07.410 --> 00:04:09.570 We want that rate of false accusations 00:04:09.570 --> 00:04:11.310 to be as low as we can get it 00:04:11.310 --> 00:04:14.250 but we're never gonna get it all the way to zero. 00:04:14.250 --> 00:04:16.500 It'll always be possible for an innocent blob 00:04:16.500 --> 00:04:19.770 to get an epic streak of heads and look suspicious. 00:04:19.770 --> 00:04:23.490 So we have to make a decision about what's good enough. 00:04:23.490 --> 00:04:25.740 The standard choice here is 5%, 00:04:25.740 --> 00:04:29.880 or one false accusation out of every 20 fair players. 00:04:29.880 --> 00:04:32.400 We could choose a different value if we wanted to, 00:04:32.400 --> 00:04:34.000 but we might as well start here. 00:04:43.200 --> 00:04:46.350 Okay, so at this point, we've crossed the threshold. 00:04:46.350 --> 00:04:50.820 There's only a one in 32 or 3.125% chance 00:04:50.820 --> 00:04:54.000 of seeing five heads in a row from a fair coin. 00:04:54.000 --> 00:04:56.700 So one possible test we could use would be, 00:04:56.700 --> 00:04:59.220 if a player gets five out of five heads, 00:04:59.220 --> 00:05:01.200 accuse them of being a cheater. 00:05:01.200 --> 00:05:03.570 Otherwise, decide they're innocent. 00:05:03.570 --> 00:05:05.880 So let's see how this test performs. 00:05:05.880 --> 00:05:07.710 We're gonna want a lot of data. 00:05:07.710 --> 00:05:09.900 So let's make a set of 1000 players 00:05:09.900 --> 00:05:11.553 where half of them are cheaters. 00:05:16.800 --> 00:05:19.800 Before we see the results, try making some predictions. 00:05:19.800 --> 00:05:22.830 How often will it wrongfully accuse fair players? 00:05:22.830 --> 00:05:25.780 And what fraction of the cheaters do you think it'll catch? 00:05:28.890 --> 00:05:32.340 Alright, we can divide these blobs into four categories. 00:05:32.340 --> 00:05:35.310 Fair players the test decided are fair. 00:05:35.310 --> 00:05:38.070 Fair players we wrongly accused of cheating. 00:05:38.070 --> 00:05:39.750 Cheaters who got away with it. 00:05:39.750 --> 00:05:42.030 And cheaters we caught. 00:05:42.030 --> 00:05:45.120 It looks like we achieved goal one with flying colors. 00:05:45.120 --> 00:05:48.480 Not only did we accuse fewer than 5% of the fair players, 00:05:48.480 --> 00:05:51.120 the test did even better than expected. 00:05:51.120 --> 00:05:53.100 When we use this test in the real world, 00:05:53.100 --> 00:05:55.380 we won't know how many fair players there are, 00:05:55.380 --> 00:05:57.750 but seeing how the test performed on this sample, 00:05:57.750 --> 00:05:59.910 combined with our analysis from before, 00:05:59.910 --> 00:06:01.650 it feels like we can be pretty confident 00:06:01.650 --> 00:06:04.263 that we would accuse fewer than 5% of players. 00:06:05.340 --> 00:06:06.990 We didn't catch very many cheaters, 00:06:06.990 --> 00:06:08.790 but that's not too surprising. 00:06:08.790 --> 00:06:10.200 We haven't even thought about them yet, 00:06:10.200 --> 00:06:12.570 so I'm sure we could do better. 00:06:12.570 --> 00:06:14.430 Before we make the next version of the test, 00:06:14.430 --> 00:06:17.850 I think it's worth mentioning some fancy statistics terms. 00:06:17.850 --> 00:06:20.490 They aren't always necessary, but you might see them around, 00:06:20.490 --> 00:06:22.170 and like any specialized words, 00:06:22.170 --> 00:06:25.800 they do make communication easier in some contexts. 00:06:25.800 --> 00:06:28.890 If a test result says not to accuse a blob of cheating, 00:06:28.890 --> 00:06:32.520 it's called a negative result, since nothing was found. 00:06:32.520 --> 00:06:35.310 And when the test does say to accuse a blob of cheating, 00:06:35.310 --> 00:06:37.260 it's called a positive result. 00:06:37.260 --> 00:06:39.630 Cheating is a bad thing, so not very positive, 00:06:39.630 --> 00:06:41.130 but the term positive here 00:06:41.130 --> 00:06:43.207 is referring to the test saying 00:06:43.207 --> 00:06:45.990 "Yes, the thing I'm looking for is here." 00:06:45.990 --> 00:06:48.150 The same is true for medical tests. 00:06:48.150 --> 00:06:49.860 If the test finds what it's looking for, 00:06:49.860 --> 00:06:51.990 the result is called positive, 00:06:51.990 --> 00:06:54.180 even though it's usually a bad thing. 00:06:54.180 --> 00:06:56.820 So we have positive and negative test results, 00:06:56.820 --> 00:06:59.670 but the results may not agree with reality. 00:06:59.670 --> 00:07:02.040 When we test a blob that's using a fair coin, 00:07:02.040 --> 00:07:04.380 the correct result would be negative. 00:07:04.380 --> 00:07:06.180 So if the test does come up negative, 00:07:06.180 --> 00:07:08.370 it's called a true negative. 00:07:08.370 --> 00:07:11.160 And if the test comes out positive, that's wrong, 00:07:11.160 --> 00:07:13.320 so it's called a false positive. 00:07:13.320 --> 00:07:14.670 And when we test a cheater, 00:07:14.670 --> 00:07:16.350 the correct result would be positive, 00:07:16.350 --> 00:07:18.240 so if the test does come up positive, 00:07:18.240 --> 00:07:20.490 we call it a true positive. 00:07:20.490 --> 00:07:23.730 And if the test incorrectly gives a negative result, 00:07:23.730 --> 00:07:25.113 that's a false negative. 00:07:26.940 --> 00:07:30.600 We can also rephrase that first goal using another new term, 00:07:30.600 --> 00:07:32.820 the false positive rate. 00:07:32.820 --> 00:07:35.880 This can be a dangerously confusing term though. 00:07:35.880 --> 00:07:38.880 It's easy to mix up what the denominator should be. 00:07:38.880 --> 00:07:41.880 False positive rate sounds like you're saying, 00:07:41.880 --> 00:07:43.200 out of all the positives, 00:07:43.200 --> 00:07:46.140 what fraction of those positives are false. 00:07:46.140 --> 00:07:50.280 Or even, out of all the tests, how many are false positives? 00:07:50.280 --> 00:07:53.250 But really, it's saying, out of all the fair players, 00:07:53.250 --> 00:07:55.953 how many of them are falsely labeled positive? 00:07:57.930 --> 00:08:00.210 I've known these words for quite a while, 00:08:00.210 --> 00:08:02.280 but my brain still automatically interprets it 00:08:02.280 --> 00:08:04.590 the wrong way basically every time. 00:08:04.590 --> 00:08:07.200 So to keep things as clear as possible for this video, 00:08:07.200 --> 00:08:09.650 we'll keep using the longer wording for goal one. 00:08:10.830 --> 00:08:13.263 Okay, let's go back to designing the test. 00:08:14.160 --> 00:08:15.600 We still need to figure out a way 00:08:15.600 --> 00:08:18.060 to achieve goal number two. 00:08:18.060 --> 00:08:21.090 Let's start by making the goal more precise. 00:08:21.090 --> 00:08:22.530 To do that, we need to pick a number 00:08:22.530 --> 00:08:25.830 for the minimum fraction of cheaters we want to catch. 00:08:25.830 --> 00:08:27.330 Using the terms from before, 00:08:27.330 --> 00:08:30.330 we could also call this the minimum true positive rate. 00:08:30.330 --> 00:08:32.550 But again, let's stick with the plain language. 00:08:32.550 --> 00:08:34.920 And to throw even more words at you, 00:08:34.920 --> 00:08:36.570 this minimum is sometimes called 00:08:36.570 --> 00:08:39.390 the statistical power of the test. 00:08:39.390 --> 00:08:42.153 It's the power of the test to detect a cheater. 00:08:43.230 --> 00:08:47.820 The standard target for statistical power is 80%. 00:08:47.820 --> 00:08:50.220 Just like the 5% number in the first goal, 00:08:50.220 --> 00:08:52.380 we could pick any value we want here. 00:08:52.380 --> 00:08:54.360 But let's run with 80% for now, 00:08:54.360 --> 00:08:57.390 and we'll talk about different choices later on. 00:08:57.390 --> 00:08:58.470 Now for calculating 00:08:58.470 --> 00:09:01.590 what we expect the true positive rate to be. 00:09:01.590 --> 00:09:02.640 What's the probability 00:09:02.640 --> 00:09:05.220 that a cheater would get five heads in a row? 00:09:05.220 --> 00:09:06.970 Take a moment to try that yourself. 00:09:11.670 --> 00:09:14.160 Okay, that was kind of a trick question. 00:09:14.160 --> 00:09:15.780 There's no way to calculate that number, 00:09:15.780 --> 00:09:17.610 since we haven't actually said anything 00:09:17.610 --> 00:09:21.390 about how often an unfair coin comes up heads. 00:09:21.390 --> 00:09:23.760 In that trial we just did with 1000 blobs, 00:09:23.760 --> 00:09:26.100 the cheaters were using coins that land heads 00:09:26.100 --> 00:09:27.960 75% of the time. 00:09:27.960 --> 00:09:30.930 We don't know for sure if that's what the real blobs do. 00:09:30.930 --> 00:09:34.050 So this 75% is an assumption. 00:09:34.050 --> 00:09:37.170 But we need some number here to calculate the probabilities, 00:09:37.170 --> 00:09:39.510 so we gotta run with something. 00:09:39.510 --> 00:09:42.990 And yet another word, this is called the effect size. 00:09:42.990 --> 00:09:47.040 In this case, it's the effect of using an unfair coin. 00:09:47.040 --> 00:09:48.450 You might be getting annoyed 00:09:48.450 --> 00:09:50.610 that this is the third time I've said 00:09:50.610 --> 00:09:53.670 we should just run with some arbitrary number. 00:09:53.670 --> 00:09:55.080 But what can I tell ya? 00:09:55.080 --> 00:09:58.470 Some things are uncertain and some things are up to us. 00:09:58.470 --> 00:10:00.210 The important thing is to remember 00:10:00.210 --> 00:10:03.150 when we're making an assumption or making a choice. 00:10:03.150 --> 00:10:04.890 That way we can note our assumptions 00:10:04.890 --> 00:10:07.800 when we make any conclusions, and we can adjust the test 00:10:07.800 --> 00:10:10.350 for different choices if we change our minds later. 00:10:11.370 --> 00:10:13.950 But now that we have a number, let's do the calculation. 00:10:13.950 --> 00:10:17.250 If the probability of each heads is 0.75, 00:10:17.250 --> 00:10:22.250 the probability of five heads in a row is 0.75 to the fifth, 00:10:22.260 --> 00:10:24.810 or about 24%. 00:10:24.810 --> 00:10:29.490 So our existing test should catch about 24% of cheaters. 00:10:29.490 --> 00:10:31.590 And hey, that is pretty close to what we saw in the trial, 00:10:31.590 --> 00:10:34.200 so everything seems to be fitting together. 00:10:34.200 --> 00:10:36.783 But our goal is to catch 80% of cheaters. 00:10:37.650 --> 00:10:40.170 The current test is a little bit extreme. 00:10:40.170 --> 00:10:44.190 It requires 100% heads for a positive result. 00:10:44.190 --> 00:10:47.100 This does make false positives unlikely, which is good, 00:10:47.100 --> 00:10:51.270 but it also makes true positives unlikely, which is bad. 00:10:51.270 --> 00:10:53.190 So we're gonna have to think about a test 00:10:53.190 --> 00:10:56.190 that allows for a mixture of heads and tails. 00:10:56.190 --> 00:10:58.230 Calculating probabilities for something like this 00:10:58.230 --> 00:11:00.150 can be a bit confusing though. 00:11:00.150 --> 00:11:02.250 For example, if we make a new test 00:11:02.250 --> 00:11:05.070 that requires a blob to flip their coin 10 times, 00:11:05.070 --> 00:11:06.690 and accuses them of being a cheater 00:11:06.690 --> 00:11:08.910 if they get seven or more heads, 00:11:08.910 --> 00:11:10.650 the calculations in that situation 00:11:10.650 --> 00:11:11.940 are gonna be a lot harder. 00:11:11.940 --> 00:11:13.290 There are a bunch of ways 00:11:13.290 --> 00:11:15.840 for there to be seven heads out of 10 flips. 00:11:15.840 --> 00:11:17.850 And we also have to think about the possibilities 00:11:17.850 --> 00:11:19.713 of eight, nine, and 10 heads. 00:11:20.610 --> 00:11:21.900 To start making sense of this, 00:11:21.900 --> 00:11:24.450 let's go back to just two flips. 00:11:24.450 --> 00:11:25.290 With a fair coin, 00:11:25.290 --> 00:11:28.650 each of these four possible outcomes is equally likely. 00:11:28.650 --> 00:11:31.290 So the probabilities are one out of four 00:11:31.290 --> 00:11:32.730 for getting zero heads, 00:11:32.730 --> 00:11:35.190 two out of four for getting exactly one heads, 00:11:35.190 --> 00:11:38.310 and one out of four to get two heads. 00:11:38.310 --> 00:11:40.200 But with an unfair coin that favors heads, 00:11:40.200 --> 00:11:42.500 they're skewed toward results with more heads. 00:11:45.150 --> 00:11:48.000 With three flips, there are eight possibilities total, 00:11:48.000 --> 00:11:50.133 with four possible numbers of heads. 00:11:52.740 --> 00:11:54.420 As we add more and more flips, 00:11:54.420 --> 00:11:56.130 it quickly becomes quite a chore 00:11:56.130 --> 00:11:58.020 to list out all the possible outcomes 00:11:58.020 --> 00:12:00.000 and add up the probabilities. 00:12:00.000 --> 00:12:01.500 But there is a pattern to it, 00:12:01.500 --> 00:12:03.990 so thankfully there's a formula for cases like this 00:12:03.990 --> 00:12:06.150 called the binomial distribution. 00:12:06.150 --> 00:12:07.620 It's not as scary as it looks, 00:12:07.620 --> 00:12:11.400 but still a full explanation deserves its own video. 00:12:11.400 --> 00:12:13.680 I'll put some links about this in the description, 00:12:13.680 --> 00:12:15.900 but for now just know that this formula 00:12:15.900 --> 00:12:17.760 is what we're using to make these bar graphs, 00:12:17.760 --> 00:12:19.620 and it follows the same pattern we used 00:12:19.620 --> 00:12:21.363 for two flips and three flips. 00:12:22.800 --> 00:12:25.530 Now let's go back to our test rule from before, 00:12:25.530 --> 00:12:26.640 where we accuse a player 00:12:26.640 --> 00:12:29.280 if they get five out of five heads. 00:12:29.280 --> 00:12:31.050 We can show the rule on these graphs 00:12:31.050 --> 00:12:32.340 by drawing a vertical line 00:12:32.340 --> 00:12:35.190 that separates the positive test results on the right, 00:12:35.190 --> 00:12:37.800 from the negative test results on the left. 00:12:37.800 --> 00:12:39.390 On the fair player graph, 00:12:39.390 --> 00:12:42.510 the bars to the left represent the true negatives, 00:12:42.510 --> 00:12:44.820 or the innocent blobs we leave alone, 00:12:44.820 --> 00:12:47.250 and to the right are the false positives, 00:12:47.250 --> 00:12:50.220 the fair players we wrongfully accuse. 00:12:50.220 --> 00:12:52.590 And on the cheater graph, the bars to the left 00:12:52.590 --> 00:12:54.480 represent the false negatives, 00:12:54.480 --> 00:12:56.730 the cheaters who evade our detection, 00:12:56.730 --> 00:12:58.980 and the bars to the right are the true positives, 00:12:58.980 --> 00:13:00.123 the cheaters we catch. 00:13:01.020 --> 00:13:03.210 Just like before, we can see that this test 00:13:03.210 --> 00:13:06.240 satisfies our first goal of accusing less than 5% 00:13:06.240 --> 00:13:09.300 of the fair players we test on average. 00:13:09.300 --> 00:13:11.070 But it doesn't satisfy our second goal 00:13:11.070 --> 00:13:13.590 of catching at least 80% of the cheaters we test, 00:13:13.590 --> 00:13:15.390 again, on average. 00:13:15.390 --> 00:13:16.560 But now that we have these graphs, 00:13:16.560 --> 00:13:19.560 we can see what happens when we change the number of heads. 00:13:19.560 --> 00:13:22.740 If we lower the threshold to four or more heads out of five, 00:13:22.740 --> 00:13:25.290 we don't meet either requirement. 00:13:25.290 --> 00:13:27.030 If we keep lowering the threshold, 00:13:27.030 --> 00:13:28.920 it can allow us to meet goal two, 00:13:28.920 --> 00:13:30.840 catching more than 80% of the cheaters, 00:13:30.840 --> 00:13:34.950 but then we accuse even more fair blobs, so that won't work. 00:13:34.950 --> 00:13:38.160 Apparently, if we want to meet both goals at the same time, 00:13:38.160 --> 00:13:40.050 we're gonna need more flips. 00:13:40.050 --> 00:13:42.180 If we put these graphs right next to each other, 00:13:42.180 --> 00:13:44.280 we can see that the blue and red distributions 00:13:44.280 --> 00:13:45.900 overlap quite a lot. 00:13:45.900 --> 00:13:47.700 So it's impossible to make a test 00:13:47.700 --> 00:13:50.850 that reliably separates fair players from cheaters. 00:13:50.850 --> 00:13:54.150 But if we increase the number of flips to, say, 100, 00:13:54.150 --> 00:13:56.190 now there's a big gap between the distributions, 00:13:56.190 --> 00:13:59.340 so it's easy to find a line that separates them. 00:13:59.340 --> 00:14:00.870 But we also have this third goal 00:14:00.870 --> 00:14:02.910 of using as few coin flips as possible, 00:14:02.910 --> 00:14:05.610 so we should try to find a happy medium somehow. 00:14:05.610 --> 00:14:07.290 Since we already have the computer set up 00:14:07.290 --> 00:14:09.540 to run the numbers, we can go back to five flips 00:14:09.540 --> 00:14:11.430 and just keep trying different thresholds 00:14:11.430 --> 00:14:12.480 with more and more flips 00:14:12.480 --> 00:14:14.643 until we find a test rule that works. 00:14:21.480 --> 00:14:23.760 It turns out that the smallest test 00:14:23.760 --> 00:14:25.440 that meets our first two goals 00:14:25.440 --> 00:14:28.170 has a blob flip its coin 23 times, 00:14:28.170 --> 00:14:29.850 and the blob is accused of being a cheater 00:14:29.850 --> 00:14:32.070 if they get 16 or more heads. 00:14:32.070 --> 00:14:34.140 That's more than I would've guessed at the start, 00:14:34.140 --> 00:14:37.293 but it's not so, so huge, so, it'll do. 00:14:38.400 --> 00:14:40.773 Alright, let's use this to test a few blobs. 00:14:53.820 --> 00:14:55.920 This blob got 17 heads. 00:14:55.920 --> 00:14:58.170 That fits our rule of 16 or more, 00:14:58.170 --> 00:14:59.460 so according to that test, 00:14:59.460 --> 00:15:01.740 we should call this blob a cheater. 00:15:01.740 --> 00:15:03.900 There is another term worth mentioning here. 00:15:03.900 --> 00:15:05.670 Assuming this blob is innocent, 00:15:05.670 --> 00:15:08.670 the probability that they'd get 17 or more heads 00:15:08.670 --> 00:15:11.220 is about 1.7%. 00:15:11.220 --> 00:15:15.600 We call this 1.7% the P value for this test result. 00:15:15.600 --> 00:15:17.550 It's kind of like a false positive rate 00:15:17.550 --> 00:15:18.843 for a single test result. 00:15:19.680 --> 00:15:20.966 Kind of. 00:15:20.966 --> 00:15:24.720 1.7% is below the 5% we set as our threshold, 00:15:24.720 --> 00:15:28.440 so according to the test, we call this one a cheater. 00:15:28.440 --> 00:15:30.330 And looking at it from the other direction, 00:15:30.330 --> 00:15:31.620 if the blob is cheating, 00:15:31.620 --> 00:15:34.860 using a coin that comes up heads 75% of the time, 00:15:34.860 --> 00:15:39.360 there's a 65% chance that they'd get 17 or more heads. 00:15:39.360 --> 00:15:42.210 Another way to say it is that they're in the top 65% 00:15:42.210 --> 00:15:44.160 of results we'd expect from cheaters. 00:15:44.160 --> 00:15:46.500 So if we wanna catch 80% of the cheaters 00:15:46.500 --> 00:15:48.300 we'd better call this one a cheater. 00:15:49.440 --> 00:15:51.483 Okay, let's try it with one more blob. 00:16:04.500 --> 00:16:07.020 This one got 13 heads. 00:16:07.020 --> 00:16:09.330 This is more than half of the 23 flips, 00:16:09.330 --> 00:16:11.580 so it's tempting to call it a cheater. 00:16:11.580 --> 00:16:14.220 But 13 is below the 16 heads 00:16:14.220 --> 00:16:17.010 the test requires for a positive result, so, 00:16:17.010 --> 00:16:18.900 we call it a fair player. 00:16:18.900 --> 00:16:21.930 The P value of this result is about 34%. 00:16:21.930 --> 00:16:24.780 So if we accuse players with results like this, 00:16:24.780 --> 00:16:28.890 we'd expect to wrongly accuse about 34% of players. 00:16:28.890 --> 00:16:31.200 That's well beyond our 5% tolerance, 00:16:31.200 --> 00:16:33.540 so we can't call it a cheater. 00:16:33.540 --> 00:16:35.760 And looking at it from the other direction, 00:16:35.760 --> 00:16:39.450 if it were a cheater, there would be about a 99% chance 00:16:39.450 --> 00:16:41.640 that they'd get this many heads or more. 00:16:41.640 --> 00:16:43.800 We don't have to catch 99% of the cheaters 00:16:43.800 --> 00:16:46.740 to hit our 80% goal, so we can still meet that goal 00:16:46.740 --> 00:16:49.290 if we let this one off the hook. 00:16:49.290 --> 00:16:51.840 Is the first one really a cheater? 00:16:51.840 --> 00:16:54.270 Is that second one really playing fair? 00:16:54.270 --> 00:16:55.410 We can't know for sure, 00:16:55.410 --> 00:16:57.390 but based on how we designed our test 00:16:57.390 --> 00:17:00.570 we should expect to catch at least 80% of the cheaters, 00:17:00.570 --> 00:17:03.993 and falsely accuse less than 5% of the fair players. 00:17:06.630 --> 00:17:08.460 So now let's see how this test does 00:17:08.460 --> 00:17:10.860 on another group of 1000 blobs. 00:17:10.860 --> 00:17:13.080 Like before, half the blobs in this group 00:17:13.080 --> 00:17:14.220 are using a trick coin 00:17:14.220 --> 00:17:17.703 that has a 75% probability of landing heads. 00:17:22.440 --> 00:17:24.780 Okay, the results do look about right. 00:17:24.780 --> 00:17:28.560 We accused less than 5% of the fair players, 00:17:28.560 --> 00:17:31.413 and we caught more than 80% of the cheaters. 00:17:32.400 --> 00:17:36.480 5% and 80% are the normal numbers for historical reasons. 00:17:36.480 --> 00:17:38.940 So we could make different decisions if we like. 00:17:38.940 --> 00:17:39.780 Maybe we decide 00:17:39.780 --> 00:17:42.270 that we really do not want to bother the blobs 00:17:42.270 --> 00:17:43.500 who are playing fairly. 00:17:43.500 --> 00:17:46.710 So we wanna lower the false positive rate to 1%. 00:17:46.710 --> 00:17:48.870 To achieve this with 23 flips, 00:17:48.870 --> 00:17:52.950 we'd have to raise the heads threshold to 18 heads. 00:17:52.950 --> 00:17:55.050 This would lower the fraction of cheaters we catch 00:17:55.050 --> 00:17:57.240 to about 47% though. 00:17:57.240 --> 00:17:59.520 If we don't want to increase the number of flips, 00:17:59.520 --> 00:18:02.310 we could decide we're okay with that 47%, 00:18:02.310 --> 00:18:04.590 maybe we just want cheating to feel risky, 00:18:04.590 --> 00:18:07.320 so 47% is good enough. 00:18:07.320 --> 00:18:10.410 Or, if we still want to catch 80% of the cheaters we test, 00:18:10.410 --> 00:18:12.090 we could increase the number of flips 00:18:12.090 --> 00:18:14.840 until we find a test that achieves both of those goals. 00:18:21.690 --> 00:18:23.100 We could also be super hardcore 00:18:23.100 --> 00:18:25.740 and go for a 99% true positive rate, 00:18:25.740 --> 00:18:27.783 and a 1% false positive rate. 00:18:32.640 --> 00:18:34.590 But we'd have to flip the coin 80 times 00:18:34.590 --> 00:18:35.640 to get to that level. 00:18:36.570 --> 00:18:39.600 We'll always be able to set two of these goals in stone, 00:18:39.600 --> 00:18:42.870 but that'll limit how well we can do on the third goal. 00:18:42.870 --> 00:18:45.090 How to set these goals depends which trade-offs 00:18:45.090 --> 00:18:46.620 we're willing to make. 00:18:46.620 --> 00:18:47.880 For the rest of this video though, 00:18:47.880 --> 00:18:52.530 we're just gonna go with the standard 5% and 80%. 00:18:52.530 --> 00:18:54.750 Now that we've settled on the goals we're going for, 00:18:54.750 --> 00:18:57.750 and we have a test that seems to be achieving those goals, 00:18:57.750 --> 00:18:59.643 let's test one more set of blobs. 00:19:00.870 --> 00:19:02.760 To pretend these are real blobs 00:19:02.760 --> 00:19:04.710 and not some artificial sample, 00:19:04.710 --> 00:19:07.200 I'm not going to tell you anything about this group 00:19:07.200 --> 00:19:09.960 except that there are 1000 of them. 00:19:09.960 --> 00:19:11.430 How do you think this test will do 00:19:11.430 --> 00:19:13.290 on this more mysterious group? 00:19:13.290 --> 00:19:17.490 Will it manage to accuse fewer than 5% of the fair players? 00:19:17.490 --> 00:19:20.640 And will it catch 80% of the cheaters? 00:19:20.640 --> 00:19:23.760 At this point in the video, it would be easy to get lazy 00:19:23.760 --> 00:19:25.950 and not actually make the predictions. 00:19:25.950 --> 00:19:28.710 But if I'm asking you these questions yet again, 00:19:28.710 --> 00:19:31.650 something must be about to go wrong, right? 00:19:31.650 --> 00:19:33.210 Or, maybe I'm just pretending 00:19:33.210 --> 00:19:34.800 so you'll engage a little more. 00:19:34.800 --> 00:19:36.360 Who can say? 00:19:36.360 --> 00:19:37.860 But really, what do you think? 00:19:42.030 --> 00:19:45.240 Okay, so we labeled about a fifth of them to be cheaters, 00:19:45.240 --> 00:19:47.550 which is a bit less than before. 00:19:47.550 --> 00:19:50.820 If this were the real world, that's all you would get. 00:19:50.820 --> 00:19:52.530 You wouldn't get to see who was really cheating 00:19:52.530 --> 00:19:54.000 and who was really innocent 00:19:54.000 --> 00:19:57.810 to get confirmation that the test is working as expected. 00:19:57.810 --> 00:20:00.480 I mean, maybe you could, but it would take more testing. 00:20:00.480 --> 00:20:03.390 You couldn't do it with this test alone. 00:20:03.390 --> 00:20:05.820 But because this is a computer simulation, 00:20:05.820 --> 00:20:08.070 I do know the full truth. 00:20:08.070 --> 00:20:10.800 This group was 90% cheaters. 00:20:10.800 --> 00:20:14.160 We still accused less than 5% of the fair players, 00:20:14.160 --> 00:20:17.490 but we only caught about a quarter of the cheaters. 00:20:17.490 --> 00:20:19.170 Something went wrong. 00:20:19.170 --> 00:20:21.180 The problem is that we assumed 00:20:21.180 --> 00:20:24.600 that the cheater coins came up heads 75% of the time. 00:20:24.600 --> 00:20:26.460 And that assumption was wrong. 00:20:26.460 --> 00:20:28.380 The real world cheaters were using coins 00:20:28.380 --> 00:20:31.170 that came up heads 60% of the time. 00:20:31.170 --> 00:20:32.790 If we knew that from the beginning, 00:20:32.790 --> 00:20:35.670 we still could have designed a test to achieve our goals, 00:20:35.670 --> 00:20:38.010 but it would need 158 flips 00:20:38.010 --> 00:20:41.610 and require 90 heads to reach those same thresholds, 00:20:41.610 --> 00:20:45.300 which is honestly way more coin flips than I was expecting. 00:20:45.300 --> 00:20:47.130 But in hindsight, it's not that surprising 00:20:47.130 --> 00:20:48.420 that we need a lot of data 00:20:48.420 --> 00:20:50.493 to tease out that smaller difference. 00:20:51.570 --> 00:20:53.070 But we didn't design that test 00:20:53.070 --> 00:20:55.830 because we got the effect size wrong. 00:20:55.830 --> 00:20:59.880 I know, I know, I was the one who said we should assume 75%. 00:20:59.880 --> 00:21:01.710 But be honest with yourself. 00:21:01.710 --> 00:21:03.150 Did you remember that assumption 00:21:03.150 --> 00:21:04.770 when making your prediction? 00:21:04.770 --> 00:21:07.980 It's very easy to forget that assumptions are assumptions, 00:21:07.980 --> 00:21:10.560 and instead just treat them as facts. 00:21:10.560 --> 00:21:13.830 This concludes me tricking you to try to teach you a lesson, 00:21:13.830 --> 00:21:16.863 but they really are easy mistakes to make in real life. 00:21:17.790 --> 00:21:20.040 On the bright side, though, our test did succeed 00:21:20.040 --> 00:21:23.610 at accusing less than 5% of the fair players. 00:21:23.610 --> 00:21:25.020 The framework we built up here 00:21:25.020 --> 00:21:27.840 isn't just good for catching unfair coins. 00:21:27.840 --> 00:21:29.340 It's the dominant framework used 00:21:29.340 --> 00:21:31.470 in actual scientific studies. 00:21:31.470 --> 00:21:34.110 To summarize we take a yes or no question. 00:21:34.110 --> 00:21:35.790 In this case, our question was, 00:21:35.790 --> 00:21:38.610 is this particular blob using a biased coin? 00:21:38.610 --> 00:21:40.290 But it could be any question. 00:21:40.290 --> 00:21:41.910 Then we come up with a model 00:21:41.910 --> 00:21:45.390 for what kinds of results we'd expect if the answer is yes, 00:21:45.390 --> 00:21:47.670 and if the answer is no. 00:21:47.670 --> 00:21:50.310 Then we come up with a test that can do a decent job 00:21:50.310 --> 00:21:52.500 of telling those two situations apart, 00:21:52.500 --> 00:21:54.150 according to the models. 00:21:54.150 --> 00:21:56.550 The details are usually a bit more complicated, 00:21:56.550 --> 00:21:59.010 since most real world systems that we wanna study 00:21:59.010 --> 00:22:01.020 are more complicated than coin flips. 00:22:01.020 --> 00:22:02.460 But most scientific studies 00:22:02.460 --> 00:22:04.560 have this framework at their core. 00:22:04.560 --> 00:22:06.120 Like I mentioned at the beginning 00:22:06.120 --> 00:22:09.390 this is called frequentist hypothesis testing. 00:22:09.390 --> 00:22:12.210 There's another method called Bayesian hypothesis testing 00:22:12.210 --> 00:22:15.210 which we'll look at in the next video in this series. 00:22:15.210 --> 00:22:16.043 See you then.