One of the least recognized, but most common, biases in decision-making is the survivorship bias, also called the success bias. 

Most how-to books, biographies, and advice trade on this bias.

A rich or famous person writes a book, hosts a podcast, or gives a lecture telling the secrets of his or her success with the implicit message of “do these five things that I did to be successful like me.”   Or a Hollywood actor gives advice during an Academy Award acceptance speech to “work hard and follow your dreams to do what I did.”

While doing these things may have led to that person’s success, it is unlikely that any reader or listener will achieve success by following this advice. The real question we should ask is not what five things led to this person’s success.  Instead, we should ask:

• What percentage of people who do these five things achieve the success that person achieved? 
• What percentage of actors who work hard and follow their dreams win Academy Awards?

By just looking at those who have survived or been successful, we can be led to false conclusions because we do not consider those who have failed.

Definition

The Survivorship Bias is the logical error of focusing on the people or things that have survived a process (or been successful) and not considering the people or things that did not survive (or failed) because these cannot easily be measured.

The survivorship or success bias happens whenever we are evaluating something based on an incomplete sample of data. 

Example 1

The classic example of the survivorship bias was during World War II when the US military was conducting countless bomber raids over Germany.  To minimize bomber losses, a statistician, Abraham Wald, was brought in to evaluate the planes returning from missions to see where the planes could be reinforced to reduce the number of planes being shot down.  The data showed that the fuselage, tips of the wings, and tail sections of the planes were full of bullet holes.  Wald’s counterintuitive advice was to reinforce the areas of the plane with the least number of bullet holes – the engines and cockpit.  The key insight was that planes that had been shot in the fuselage, tips of the wings, and tail sections could survive the mission and return while the planes that had been shot in the engines and cockpit did not survive the mission and thus were not considered part of the sample being evaluated.

Example 2

The second example comes from finance.  We often see impressive 5-, 10-, or 20-year average returns from a company’s line-up of mutual funds.  Since poor performing mutual funds are often shut down or merged into a successful fund, we do not get the real average of the company’s mutual fund line-up, only the average of those funds good enough to have survived 5, 10, or 20 years. 

Example 3

This last example of survivorship bias is known as the Publication Bias.  In medicine and the sciences, studies that are not successful are shut down (in medicine) or not published (in most sciences).   When meta-studies evaluating many different studies in a field are conducted, they can only evaluate the data accessible to them (the published studies).  This has happened many times in pharmaceuticals and psychology where meta-analysis of published data showed a highly effective result for a drug or treatment.  Later, researchers were able to gain access to all internal company data (including failed trials).  Analyzing all the data led to a result that was much smaller or even non-existent.

Conclusion

With survivorship (or success) bias, we consider only the winners – those who have survived or had success.  By not taking the losers into account and considering the base rate – what percent of people who make this decision or pursue this strategy are successful? – we may make riskier decisions or pursue overhyped strategies with a low probability of success.

To avoid survivorship bias, we need to constantly be looking for the losers.  Are they considered in the data we are using to make our decision?  Instead of looking at the data presented to us, we need to look at what was removed from the sample of data before it came to us.  We need to ask… Where are the Missing Bombers?