The Answer Might Surprise You!

I’m not sure if this should be a question or a post, but a post allows me to use LaTeX and this is educational in nature so I‘m posting it as a post, but moderators, please let me know if I should take this down and repost as a question (Though I will be sad because I spent way too much time on working with the TeX formatting).

Case: Disease X affects 1 in 1000 people. One day you hear that a new test for the disease comes out, and it boasts a clinically validated accuracy of 91% (sensitivity of 90% and specificity of 91%). You take the test and receive a positive test result.

Question: What is the probability that you have the disease?

Answer below, but try to answer it first and let me know In the comments! Don’t worry about getting an exact number, just estimate and give the nearest 10% (30%, 40%, 90%, etc). I’m going to go through an explanation first. For an amazing video that was the inspiration for this post and which explains things visually much bettter than anything I can write here, please check it out here.

Let’s imagine we have a random sample of 1000 people.

Has Disease	Doesn’t Have Disease
10	990

Now let’s imagine we tested each person in the sample with the test, and let’s display the results here according to the numbers given in the case.

To do this, we need to understand what the numbers in the case mean. Let’s break down the definitions!

$Sensitivity = \frac{TP}{TP+FN}$

Sensitivity = TP / (TP + FN)

Sensitivity measures how well a test can correctly identify positive cases out of all actual positive cases. It's the ability of the test to detect the presence of a condition when it's truly present. High sensitivity indicates a low rate of false negatives, meaning the test is good at catching true positives.

$Specificity = \frac{TN}{TN+FP}$

Specificity = TN / (TN + FP)

Specificity measures how well a test can correctly identify negative cases out of all actual negative cases. It's the ability of the test to correctly rule out the presence of a condition when it's truly absent. High specificity indicates a low rate of false positives, meaning the test is good at avoiding incorrect positive identifications.

$False Positive Rate (FPR)=\frac{FP}{TN+FP}$

False Positive Rate (FPR) = FP / (TN + FP)

False Positive Rate measures the proportion of actual negative cases that are incorrectly identified as positive. It's the rate of false alarms or false positive errors. A low FPR indicates that the test is good at avoiding false positives.

$False Negative Rate (FNR)=\frac{FN}{TP+FN}$

False Negative Rate (FNR) = FN / (TP + FN)

False Negative Rate measures the proportion of actual positive cases that are incorrectly identified as negative. It's the rate of missed opportunities or false negative errors. A low FNR indicates that the test is good at avoiding false negatives.

$Prevalence = \frac{TP+FN}{TP+TN+FP+FN}$

Prevalence = (TP + FN) / ( TP + TN + FP + FN)

Prevalence is the proportion of the population that has a condition. Think the number of people with it over the total number of people in the population.

$Accuracy = \frac{TP+TN}{TP+TN+FP+FN}$

Accuracy = (TP + TN) / ( TP + TN + FP + FN)

Accuracy represents the overall correctness of a test by considering both true positives and true negatives. It measures the proportion of all assessments (positive and negative) that are correct. While accuracy gives an overall picture of test performance, it may not be sufficient when dealing with imbalanced datasets or when the costs of false positives and false negatives are significantly different.

Accuracy can also be calculated by compositing the other elements, such as prevalence, sensitivity, and specificity.

$Accuracy = Sensitivity * Prevalence + Specificity * (1 - Prevalence)$

Accuracy = Sensitivity * Prevalence + Specificity * (1 - Prevalence)

So which is the best metric that actually tells us anything about the power of the test at determing whether a positive result is meaningful? Well we want a metric that compares the proportion of those with the condition to the proportion of those with a positive test result: The Positive Predictive Value! Now if we got a negative test result and we wanted to know how meaningful that result was, we would use the Negative Predictive Value.

$Positive Predictive Value (PPV)=\frac{TP}{TP+FP}$

Positive Predictive Value (PPV) = TP / (TP + FP)

Positive Predictive Value measures the probability that a person with a positive test result actually has the condition. It's the proportion of true positives out of all positive test results. A high PPV indicates a low rate of false positives.

$Negative Predictive Value (NPV)=\frac{TN}{TN+FN}$

Negative Predictive Value (NPV) = TN / (TN + FN)

Negative Predictive Value measures the probability that a person with a negative test result truly does not have the condition. It's the proportion of true negatives out of all negative test results. A high NPV indicates a low rate of false negatives.

Like accuracy, we can calculate these values as composites of the other elements.

$PPV = \frac{Sensitivity * Prevalence}{Sensitivity * Prevalence + (1 – Specificity) * (1 – Prevalence)}$

$NPV = \frac{(1 - Sensitivity) * (1 - Prevalence) }{ (1 - Sensitivity) * (1 - Prevalence) + Specificity * Prevalence}$

Now going back to the example with the 1000 people, let’s fill out our table.

We can check our metrics now:

$Sensitivity = \frac{9}{10}$

$Specificity = \frac{901}{990}$

$FPR = \frac{89}{990}$

$FNR = \frac{1}{10}$

$Prevalence = \frac{10}{1000}$

$Accuracy = \frac{910}{1000}$

$PPV = \frac{9}{98}$

$NPV = \frac{901}{902}$

So based on our calculations, the probability that we actually have disease X is 9/98 or about 11%!

How is it possible that we would only have such a small probability given that the test is over 90% accurate? Well this case is called the Medical Test Paradox, only it isn’t a true paradox because while those are logical impossibilities, this is only an apparent paradox created by the unintuitive nature of the this test.

The sensitivity and specificity tell us about the value of the test, but they don’t really tell us about it in context. Again, the sensitivity is telling us out of all those with the condition, how many were properly identified. That’s different than asking, out of those who tested positive, how many actually have it! The sensitivity and specificity are not alone taking into account the prevalence of the condition. And here’s the bingo!

The prevalence is key here. You might be thinking to yourself, “If this test is that useless, why would I take it?” Well to a random person the test IS rather useless (arguably), but when used in conjunction with other details, like the presentation of symptoms, that increases the prevalence by decreasing the population size, relatively.

What do I mean? Think of it this way. The prevalence being so small means we cannot attribute much of the test’s result is obfuscated by noise. If we increase the prevalence, either increasing the people with the condition or decreasing the size of the pool we are drawing from, then the test result has more predictive power. How do we increase the prevalence? Well let’s say that people with disease X present with symptoms like curly purple hair and a red squeaky nose, and you are presenting with these symptoms when you arrive to the doctor.The pool of people that show these symptoms are much smaller than the entire population, and so the prevalence being the proportion of those with the condition would go up as the the size of the population has decreased to only those that look like clowns!

This was such a counterintuitive question for me to tackle when I first watched this video ages ago. And years later, I still get tripped up with this! I cannot recommend the 3Blue1Brown video that was the inspiration for this post enough.

Anyways, I hope you all enjoyed this little read, and I look forward to reading your responses down below, as well as any other statistical paradoxes you’ve got for me! Also if I should edit anything or make it more clear, please let me know and I’ll go ahead (will likely do this as I read through comments anyways to make sure that this is as informative for people new to this as possible, and will give credit where credit is due!)

Happy Monday!

-Blue

Preprint

Preprint

The Answer Might Surprise You!

The Answer Might Surprise You!