Why Strategies Fail
Most strategies that look good in backtests fail live. Learn the main causes, from overfitting and costs to regime changes, and how to guard against each.
Many traders have built a strategy with a beautiful backtest, started trading it and watched it lose money. This is the norm, not the exception. Studies of published trading strategies and of retail trader results show that most apparent edges disappear in real trading. The reasons are well known and largely preventable. Understanding them is the best way to build strategies that have a chance of surviving contact with live markets.
The main causes#
| Cause | What goes wrong | Lesson |
|---|---|---|
| Overfitting | The strategy fits noise in historical data | Overfitting and Curve Fitting |
| Multiple testing | The best of many tests looks good by chance | P-Hacking and Multiple Testing |
| Look ahead bias | The backtest used information not available at the time | Look-Ahead Bias |
| Survivorship bias | Data excluded failed companies or delisted assets | Survivorship and Selection Bias |
| Data leakage | Information from the test period leaked into training | Data Leakage |
| Unrealistic costs | Commissions, spreads, slippage and borrow costs ignored | Costs and Slippage in Backtests |
| Unrealistic fills | Assuming trades fill at prices that were not available | Fill Models, Partial Fills and Order Queues |
| Capacity limits | Returns vanish at real trading size | Alpha Capacity and Crowding |
| Regime change | Market conditions differ from the test period | Structural Breaks and Regime Changes |
| Edge decay | Competition removes the edge | Signal and Alpha Decay |
| Execution and operational errors | Bugs, data outages, wrong orders | Alerts, Error Handling and Reconnection |
| Trader behaviour | Overriding rules, abandoning in drawdowns | Discipline |
A typical failure story#
The gap between backtest and live#
Researchers have documented large drops from backtest to live performance:
- Academic anomalies lose a substantial share of returns after publication (McLean and Pontiff, 2016).
- Quantopian research (2016), analysing hundreds of user strategies, found that backtest Sharpe ratios had little ability to predict out of sample performance, and that strategies tested more times showed bigger gaps.
Guarding against failure#
- Start with an economic rationale. See The Trading Research Process.
- Keep strategies simple with few parameters.
- Record every test and adjust expectations for the number of trials.
- Model costs conservatively, then add a margin of safety.
- Validate out of sample and with walk forward tests. See Walk-Forward Analysis.
- Stress test across regimes and parameter changes. See Robustness and Stress Testing.
- Start small live and scale only when results match expectations.
- Set rules for reducing or stopping before you start.
Expect lower live performance#
A practical rule of thumb used by many quant traders is to expect live performance to be well below backtest performance, sometimes cutting the backtest Sharpe ratio by half or more. If the strategy is still attractive under that assumption, it may be worth trading.
When a strategy fails, learn from it#
Review failures systematically: was it overfitting, costs, execution, regime or behaviour? Each answer improves the next research cycle. See Post-Trade Analysis.
Frequently asked questions#
Why do backtested strategies fail in live trading?#
Mainly because of overfitting, testing many variations, look ahead and survivorship bias, unrealistic costs and fills, regime changes and execution problems.
How much worse is live performance than backtests?#
It varies, but many practitioners expect live risk adjusted returns to be substantially lower, often half or less of backtested figures.
How can I make my strategy more likely to work live?#
Use a clear rationale, keep rules simple, model costs conservatively, validate out of sample, start small and monitor results against expectations.
Next, learn how to build a reliable backtest in Backtesting Methodology.
3 quick questions on this lesson. Get them all right to finish it.
Turn on JavaScript to take the quiz.
Mentioned in
- The Trading Research ProcessResearch and Backtesting
- The Strategy LifecycleResearch and Backtesting
- Quant Trading Learning PathStart Here
- Sunk Cost FallacyTrading Psychology
- From Backtest to Live: Paper, Shadow and CanaryAlgorithmic Trading