Survivorship and Selection Bias

Survivorship bias happens when you only look at things that survived and ignore those that failed. Selection bias, more broadly, happens when the sample you study is not representative of what you want to learn about. In trading, both quietly inflate results: backtests that ignore delisted stocks, fund databases that drop closed funds and success stories from traders who happened to be lucky. Recognising these biases protects you from false confidence in strategies, funds and advice.

Survivorship bias in backtests#

If a stock database contains only companies that exist today, it excludes every company that went bankrupt, was acquired or was delisted. Those companies often had poor returns before disappearing. A backtest on survivors overstates returns, especially for strategies that buy beaten down or small stocks.

Survivorship in fund performance#

Fund databases often drop funds that close, usually after poor performance. Studies have estimated that survivorship bias inflates average reported hedge fund returns by roughly 1 to 3 percentage points a year, depending on the database and period. Mutual fund studies show similar effects. Backfill bias adds to this: funds join databases after a good track record, and their earlier returns are added retrospectively. See Hedge Funds.

Selection bias in trading advice#

If thousands of people trade randomly, some will have spectacular records by chance. Without knowing how many tried, a great record says less than it seems. See Fake Performance and Track Record Verification and Identifying Trading Scams.

Selection bias in research#

• Choosing the period: starting a backtest right after a crash makes almost any long strategy look good.
• Choosing the market: testing an idea on many markets and reporting only the ones where it worked.
• Dropping inconvenient data: excluding "unusual" periods such as 2008 or 2020.
• Publication bias: academic journals tend to publish positive findings, so published anomalies overstate the success rate of ideas. See P-Hacking and Multiple Testing.

How to avoid these biases#

1. Use survivorship free data, including delisted and merged securities with their final returns.
2. Use historical index memberships for universe selection.
3. Test over full periods, including crises.
4. Report all markets and periods tested, not just the good ones.
5. Be sceptical of track records without independent verification and context about how many others tried.
6. Ask what is missing from any dataset or story.

The WWII bombers story#

A famous illustration comes from statistician Abraham Wald during the Second World War. Analysts studied damage on returning bombers to decide where to add armour. Wald pointed out that the planes that were hit in other places did not return; armour should go where the surviving planes showed no damage. The lesson for traders: the data you can see may be exactly the wrong data to learn from.

Frequently asked questions#

What is survivorship bias in trading?#

Ignoring assets, funds or traders that failed and disappeared, which makes historical results look better than they were.

How does survivorship bias affect backtests?#

Backtests using only currently listed stocks miss companies that went bankrupt or were delisted, overstating returns, especially for small cap and contrarian strategies.

What is selection bias?#

Drawing conclusions from a sample that is not representative, such as only studying successful traders or choosing favourable test periods.

Next, learn about subtler leaks of information in Data Leakage.