Confoundle · a reasoning trap

Survivorship bias

It's easy to study the winners, the survivors, the successes, the things still standing, and copy what they have in common. But the failures are invisible: they dropped out of the data. Whatever helped the survivors survive looks far more powerful than it is, because you never see everyone it didn't save. Before copying the winners, ask who's missing.

The rule

When you only look at the winners, the failures become invisible, and they often hold the real lesson.

What it looks like

Bombers come home riddled with bullet holes. Where do you add the armour?In WWII, returning bombers were peppered with damage, heaviest on the wings and body, while the engines and cockpit came back almost untouched. Armour is heavy, so you can only reinforce a few areas.
Armour where the holes aren't.These are the planes that made it home. The ones hit in the engine or cockpit didn't, so their damage never shows up in the data. The holes on the survivors map out exactly where a bomber can be shot and still fly. The clean spots are the fatal ones: armour those.

Why it works

Survivorship bias creeps in whenever your data has quietly been filtered to keep only the things that “made it”: returning planes, funds still trading, companies still around. You never see the ones that failed and dropped out, and because the survivors share whatever helped them survive, that trait looks far more common, or more effective, than it really is. The fix is to hunt for the missing group and ask what the full picture would show. (The real Wald did more than point at a diagram: he built a statistical method to estimate each part's vulnerability from the survivors' damage.)

Source

Wald A. A Method of Estimating Plane Vulnerability Based on Damage of Survivors. Statistical Research Group, Columbia University, 1943 (reprinted by the Center for Naval Analyses, 1980). The “armour the clean spots” story popularises Wald's actual statistical method.

See if it fools you

This page gives the answer away. The puzzle version shows you the same figures first and asks you to commit before the reveal.

Play this one