Confoundle · a reasoning trap

The base-rate fallacy

A test can be 95% accurate and a positive result can still mean you're almost certainly fine. The trick is how rare the thing is. If only 1 in 1,000 people have a disease, then among everyone who tests positive, the few real cases are buried under a pile of false alarms. Accuracy isn't the same as your actual odds; you have to ask how common it is first.

The rule

When something is rare, even a very accurate test throws up far more false alarms than real cases, so a positive result can still mean you're probably fine.

What it looks like

A near-perfect test says you're sick. How worried should you be?This disease is rare, about 1 in 1,000 people have it. The test never misses it when it's really there, and it raises a false alarm on only about 1 in 20 healthy people. Your result just came back positive.

Positive, but almost certainly a false alarm.Because almost nobody has the disease, the test's small error rate does the heavy lifting. In 1,000 people, only 1 is truly sick, but about 50 healthy people also get a positive. So among the ~51 positive results, just 1 is real. A positive barely nudges you from “very unlikely” to “still unlikely.”

Why it works

A test's accuracy and your actual odds are two different things. Accuracy is measured on people we already know are sick or healthy. But a positive result asks the reverse question (given this positive, am I sick?), and that depends on how many sick people there were to find in the first place. If only 1 in 1,000 has the disease, the huge healthy majority produces a flood of false alarms that swamps the single real case. Make the disease common and the same test looks excellent; make it rare and a positive means little on its own.

Source

Casscells W, Schoenberger A, Grayboys TB. Interpretation by physicians of clinical laboratory results. N Engl J Med. 1978;299(18):999-1001. (Prevalence 1/1000, 5% false-positive rate; the modal physician answer was 95%, correct ≈ 2%.)

See if it fools you

This page gives the answer away. The puzzle version shows you the same figures first and asks you to commit before the reveal.

Play this one