Survivor Bias: Why Studying Winners Gives You the Wrong Answer

Every list of what successful people do is missing the same thing. Nobody surveys the people who did all of it and failed, and by the time you’re reading the list you can’t tell they were ever there.

Read any account of how somebody succeeded and you’re reading a sample of one, chosen because it worked out. That’s fine as far as it goes. The trouble starts when we treat the collection of those accounts as though it described everybody who tried.

The people who did the same things and failed are harder to find, less pleasant to interview, and they don’t come to the conference. The result is that we build our entire picture of what works out of the ones who made it through, and we rarely notice the sample was assembled for us before we arrived.

This has a history, and it has arithmetic you can check. Once you can run the math you stop being persuaded by numbers that sound overwhelming and mean nothing.

What is survivor bias?

The clearest illustration comes out of World War II. Abraham Wald (Mangel & Samaniego, 1984) worked with the Statistical Research Group at Columbia on a problem the Army Air Forces needed solved: where do you put armor on a bomber?

Armor is heavy, so you can’t armor everything. The obvious approach is to look at the planes coming back, map where they took hits, and reinforce the places with the most damage. Wings and fuselage came back full of holes. Engines came back comparatively clean.

Wald noticed something about the sample. Every plane they were measuring had made it home. Damage in those places was survivable, and the plane sitting in front of you being measured is the proof. The clean areas were clean because a plane hit there didn’t come back to be part of the sample.

(The popular version of the story is tidier than the history. Wald wrote a series of technical memoranda on estimating vulnerability from selectively observed survivors, and the internet has compressed that into one dramatic moment in a briefing room. The mathematics are real.)

The missing planes carried the information. Everything below is a version of the same problem.

Why does this fool intelligent people?

Three habits of thought on this.

Examples that come to mind easily feel common, whether or not they’re. Once you know how a story ended, the road to that ending looks planned and obvious, even when the person living it was guessing. And we convert messy, partly random histories into clean stories with a hero and a turning point, because a story is far easier to tell than a pile of numbers.

None of this requires anybody to lie. Someone who built something great can sincerely believe one principle caused it, while underweighting timing, capital, a market that moved their way, a competitor who stumbled, one extraordinary early hire, and plain luck. Their account is sincere testimony about their experience. Whether it explains the cause is a separate question.

What does the arithmetic look like?

Here’s the smallest example that shows it. Take 1,000 people who tried the same thing. One hundred succeeded, nine hundred didn’t.

The claim: 80 of the 100 successful people used approach X. That’s 80%, it’s true, it’s verifiable, and it’ll get a standing ovation at any conference.

Now run it the other way. Of the 880 people who used X, 80 succeeded, which is 9.1%. Of the 120 who didn’t use X, 20 succeeded, which is 16.7%. The most common behavior among winners comes attached to roughly half the success rate. There’s more evidence for avoiding X than for adopting it.

Nothing in the original claim was false. The denominator was missing, and the denominator was the whole answer.

If you only pay attention to success, you’re giving yourself misinformation.

“But I know someone who did it.” What’s wrong with that?

The famous-dropout argument. Pointing at Gates, Jobs and Zuckerberg establishes that leaving college is compatible with extraordinary success. It can’t tell you the odds, because the argument never counts the dropouts you haven’t heard of, and you haven’t heard of them precisely because it didn’t work. The cases are memorable for exactly the reason they’re useless as a guide.

“We never wore helmets and we turned out fine.” The people who didn’t turn out fine aren’t in the conversation. The outcome itself decides who gets to speak. That makes it the least reliable evidence there is, and unfortunately the most persuasive.

“They don’t make them like they used to.” A house built in 1900 and still standing has survived 126 years of weather, maintenance decisions, redevelopment pressure and economics. The badly built houses (or exactly the same build of houses that caught on fire, happened to be poorly maintained, or were abandoned, or any of the other deterioration possibilities) of 1900 were pulled down decades ago. You’re comparing the toughest survivors (or random survivors) of one era against everything currently being built in another, and those two groups were assembled by completely different processes.

Now put a number on the most common version of this, the one that starts with quit your job and go all in.

Take 10,000 people who do exactly that. U.S. Bureau of Labor Statistics data on new establishments (U.S. Bureau of Labor Statistics, 2024) puts five-year survival near half, so call it 5,000 gone by year five. Of the 5,000 still operating, most are grinding out a living rather than producing a story anybody wants to hear. The people writing the book, recording the podcast and selling the course come from a thin slice of the remainder.

The person you’re listening to is one observation from that thin slice. The other 9,000-odd are part of the same experiment and none of them are available for comment. When somebody tells you what it takes, they’re describing the only path they can see, which is the one that happened to them.

Has this fooled people who should have known better?

In Search of Excellence (1982). Peters and Waterman selected 43 excellent American companies and derived the attributes they shared (Peters & Waterman, 1982). Within about two years BusinessWeek ran a cover story titled “Oops! Who’s Excellent Now?” A large share of the 43 had run into serious trouble and Atari had effectively collapsed.

Twenty years later Tom Peters wrote a piece in Fast Company called “Tom Peters’s True Confessions.” (Peters, 2001) In it he describes how the list actually came together. Colleagues at McKinsey and other smart people were asked which companies were doing interesting work, and the quantitative measures got assembled afterward. His line was “Okay, I confess: We faked the data.” He has since said that phrasing was sharper than he meant, and that varying research measures isn’t the same as fabricating numbers. That correction is fair. The selection was still a room of people naming companies they admired.

Built to Last (1994). Collins and Porras surveyed hundreds of CEOs to identify 18 visionary companies and wrote up their shared habits (Collins & Porras, 1994). Within roughly a decade much of the list had slipped badly, Motorola, Ford, Sony, Disney, Boeing, Nordstrom and Merck among them.

Good to Great (2001). Collins selected 11 companies whose stock returns had already been exceptional and catalogued what they had in common: humble leadership, discipline, focus (Collins, 2001). Circuit City filed for bankruptcy in 2009. Fannie Mae went into federal conservatorship in 2008.

The Millionaire Next Door (1996). Stanley and Danko interviewed hundreds of millionaires and reported the habits they found: live below your means, drive a used car, invest steadily (Stanley & Danko, 1996). The advice is sound. What nobody counted is the far larger group who did all three and never got there. Nassim Taleb added a second filter in Fooled by Randomness (Taleb, 2001): the sample also caught people who happened to invest through one of the strongest bull markets in history.

The 10,000-hour rule (2008). Gladwell (2008) popularized it from research on elite performers, and the underlying finding holds up: people at the top of demanding fields have accumulated enormous practice. What it can’t tell you is how many people accumulated the same hours and never got close, because that research only included the ones who did. When Macnamara, Hambrick and Oswald ran a meta-analysis across many fields in 2014 (Macnamara et al., 2014), deliberate practice accounted for about 26% of the variation in games, 21% in music, 18% in sports, 4% in education, and under 1% in professions.

Read that last figure twice. In professional work, practice hours explained less than one percent of the difference between people. Practice still matters and you should still do it. It’s nowhere near the whole story, and the reason a generation believed it was the whole story is that the research only ever looked at the winners.

Phil Rosenzweig named the error in The Halo Effect (Rosenzweig, 2007): the delusion of connecting the winning dots. He also answers the obvious defense, which is that these were serious authors with real research budgets. If the sample is chosen by the outcome, more data doesn’t repair it. Jerker Denrell showed the same thing formally (Denrell, 2003), and added a consequence worth carrying: because failures are undersampled in the stories managers learn from, we consistently overrate risky, distinctive strategies. Those are exactly the strategies that stand out among survivors.

Enormous budgets, careful authors, real data, and the same broken first step every time. If it can happen to McKinsey partners and Stanford professors with a research team, it’ll happen to you across a lunch table.

What about a long, verifiable track record?

Line up 1,024 advisors and assume every single one of them is guessing. No skill whatsoever. Pure coin flips.

Year one, half of them happen to be right. Year two, half of those. Keep going.

At year ten, one advisor holds a flawless ten-year record built out of nothing at all. He isn’t lying and the record checks out year by year. It’s still worthless as evidence of skill, because a room of pure guessers was guaranteed to produce roughly one of him.

The error isn’t believing the record. The error is never asking how many people entered the contest.

This is why serious historical analysis in finance requires survivor-bias-free databases. Funds that perform poorly get merged or shut down and vanish from the record, so a database of surviving funds makes the whole category look better than it ever was. Elton, Gruber and Blake measured that gap directly (Elton et al., 1996) and put it at roughly 1.4% a year of overstated performance, widening the longer the study period runs. Most industries have no equivalent correction and no equivalent database.

Does this happen in science too?

A Cochrane methodology review (Hopewell et al., 2009) found that trials with positive findings had close to four times the odds of publication compared with negative or null results (odds ratio 3.90, 95% CI 2.68 to 5.68). Put in plainer terms, if 41% of negative trials get published, you’d expect roughly 73% of positive ones to make it. Positive results also reached print faster, typically four to five years against six to eight.

The literature is the surviving portion of the research. What didn’t make it through was disproportionately the studies that found no effect. Anyone reading only published work is reading a filtered sample, and it’s filtered in the direction that makes things look like they work.

The same problem runs inside individual studies. Analyze only the patients who completed a treatment and you may be describing a group that improved partly because the ones it wasn’t working for stopped coming. The direction of the error depends entirely on why the data went missing, which is why the serious question is never how much is missing but why.

Does social media make this worse?

Picture 50,000 people trying the same tactic. Perhaps 200 get extraordinary results, a couple of thousand get modest ones, and the rest get nothing worth mentioning. Of the 200, maybe 50 post about it, and a handful of those get noticed.

You now see several compelling demonstrations of a tactic and no denominator at all. Repeat that a few hundred times and you end up with a confident, evidence-backed feeling that the tactic is common, easy and reliable. That feeling was assembled entirely out of the rarest outcomes the tactic has ever produced.

The mechanics are the same in wealth, fitness, dating, publishing, investing, parenting and health. Extraordinary outcomes are both more visible and more profitable to publish, so the business model of attention amplifies exactly the observations most likely to wreck your sense of the odds.

Does this mean success is just luck?

High performance usually requires real skill, effort, judgment and persistence. Winners generally did a great deal. The claim is narrower: when many capable people compete and the outcome has any random element, the group at the very top will contain more than its share of people who got fortunate on top of being good.

There’s a stranger effect worth knowing about, because it makes survivor data produce conclusions that feel backwards. Suppose reaching the top requires both skill and luck. Among the people who got there, somebody with less skill must have had more luck, and somebody with enormous skill could get there with less. Look only at that group and skill and luck can appear unrelated or even inversely related, even if they were completely independent in the population that started.

This is why you can meet brilliant operators in awful circumstances and mediocre ones in comfortable circumstances, and conclude from the pairing that ability barely matters. Ability matters. You’re looking at a group that was assembled by an outcome, and that outcome had different paths to get there.

The productive question is neither “what is the winner’s secret” nor “was it all luck.” It’s which parts of this repeat across many people, and which parts were one-time conditions that can’t be reproduced.

Should you ignore successful people?

People who have just learned about survivor bias sometimes swing to dismissing every success story. That’s worse (and lazier) than where they started.

A survivor shows you a thing can be done. They’ll show you operational detail you’d never have invented, combinations you had not considered, and ideas worth testing. What they can’t give you is the odds. A single case tells you something is possible. Only a denominator (examining the failures) tells you how likely it’s.

Compare instead of collecting. Use the winners to generate ideas, then go looking for the people who did the same thing and didn’t make it. If the comparable failures didn’t do it, you’ve something worth acting on. If they did it just as often, you’ve found a habit of successful people rather than a cause of success.

What do you actually ask?

1. How many times have you produced this result, in how many different situations, with different people running it? Once, in one place, with you at the controls, is a story. This is why I’d listen to people who, for example, ran multiple versions of the same business in different locations, demographics, employee types.
2. How many people have you taught this to, and what happened to all of them? Not the successes. All of them.
3. Did you do this before the result, or notice it afterward? Practices spotted in hindsight get promoted to causes without ever earning it.
4. What has to be true about my situation for this to be replicatable? “It works for everybody” claims the practice is independent of market, timing, capital and person. Almost nothing is.
5. What would I see if this advice were worthless? Ask it and watch what happens.

The fifth is the strongest and the least asked. An explanation that can’t be wrong is a weak one. It’ll absorb whatever result you get, it’ll never be revised, and you’ll spend years assuming the failure was yours.

Where to start

Find the graveyard – the remains of the unsuccessful. In a business it’s the cancelled accounts, the campaigns that ran once, the products that were discontinued, the people who left. In research it’s the studies that found nothing. In your own life it’s the attempts you stopped mentioning. Most people and most organizations delete all of it (or at least it doesn’t become something paid attention to), which destroys the only comparison group they’ll ever get for free.

Then ask the five questions to the next person who tells you how something is done. Including me.

Everything behind this article, including the statistics, the psychology, the causal logic and an audit you can run on your own organization, is in the complete report. It is a free PDF you can download and keep.

If you run a business and want this applied to one, I wrote a version for owners that goes through where it hides in the numbers you already have, and an earlier piece on whose advice to take.