Statistical Significance in Trading

A strategy that made money in a backtest or a few months of live trading may have a real edge, or it may have been lucky. Statistical significance is a way of judging how unlikely the results would be if there were no edge at all. It is not proof, and it can be badly misused, but it provides a disciplined first filter. This lesson brings together the ideas from earlier lessons into practical rules for judging trading results.

The t statistic as a quick guide#

For average returns:

t = mean return / (standard deviation / √n)

For an annualised Sharpe ratio measured over T years, a convenient approximation is:

t ≈ Sharpe ratio × √T

Sample size matters more than you think#

Statistical significance depends heavily on the number of independent observations. Strategies that trade rarely, such as a few times a year, may never produce enough trades to be statistically convincing within a reasonable time. High frequency strategies accumulate observations quickly, which partly explains why their edges can be measured more precisely. See Sampling and Standard Error.

Multiple testing changes everything#

If you test many ideas, some will look significant by chance. With 20 independent tests of strategies with no edge, the chance that at least one shows p < 0.05 is 1 minus 0.95^20 ≈ 64%.

Ways to adjust:

See P-Hacking and Multiple Testing.

Significance is not importance#

• Statistical significance says an effect is unlikely to be zero.
• Economic significance says it is large enough to matter after costs, risk and capacity.

A strategy with t = 5 and an edge of 0.02% per trade may be worthless after costs, while a strategy with t = 1.5 but a strong economic rationale might deserve further study with small size. See Costs and Slippage in Backtests.

Beyond p values#

A practical checklist#

1. How many independent trades or periods?
2. What is the t statistic or confidence interval? See Confidence Intervals.
3. How many variations were tried?
4. Does it survive costs and realistic execution?
5. Is there a sensible reason it should work?
6. Does it hold out of sample?

Frequently asked questions#

What does statistically significant mean for a trading strategy?#

That its results would be unlikely if the strategy had no real edge, usually judged by a t statistic or p value.

How many years of data do I need to prove a strategy?#

It depends on the Sharpe ratio: roughly (2 / Sharpe)² years for a t statistic of 2, so 16 years for a Sharpe of 0.5 and 4 years for a Sharpe of 1.0.

Is statistical significance enough to trade a strategy?#

No. Results must also be economically meaningful after costs, robust, tested out of sample and adjusted for how many ideas were tried.

Next, learn to model relationships in Regression Analysis.