Confidence Intervals

A backtest reports a single number for average return, win rate or Sharpe ratio. But that number is an estimate from a limited sample, and the true value could be higher or lower. A confidence interval expresses this uncertainty as a range of plausible values. Reading results as ranges rather than single numbers is one of the simplest ways to avoid overconfidence in trading research.

The basic formula#

For a mean, using the normal approximation:

confidence interval = estimate ± z × standard error

For small samples, the t distribution replaces z, giving slightly wider intervals. See Student's t-Distribution and Sampling and Standard Error.

Worked example: average return#

Worked example: win rate#

CI for a proportion = p ± z × √(p × (1 - p) / n)

A 58% win rate on 50 trades: SE = √(0.58 × 0.42 / 50) ≈ 7.0%. The 95% interval is about 44% to 72%. A win rate that looks strong could plausibly be below 50%. For small samples or extreme proportions, the Wilson interval is more accurate.

What a confidence interval means#

A 95% confidence interval means that if you repeated the sampling process many times, about 95% of the intervals built this way would contain the true value. It does not mean there is a 95% probability that this particular interval contains the true value, though in practice many people use it that way. Bayesian credible intervals do allow that interpretation. See Bayesian Statistics.

Confidence intervals for the Sharpe ratio#

When normal intervals are unreliable#

• Skewed or fat tailed returns: normal intervals can be too narrow. Bootstrapping is often better. See Bootstrap and Permutation Tests.
• Autocorrelated returns: standard errors are understated; use adjusted methods. See Autocorrelation and Partial Autocorrelation.
• Small samples: use the t distribution or bootstrap.
• Data mining: if you tested many strategies and picked the best, its interval is too optimistic. See P-Hacking and Multiple Testing.

Using intervals in decisions#

1. Report intervals with every key statistic in research.
2. Look at the lower bound: would the strategy still be worth trading at the pessimistic end?
3. Compare strategies by overlap: heavily overlapping intervals mean you cannot tell which is better.
4. Size positions with the uncertainty in mind: a wide interval suggests smaller size. See Position Sizing.

A note on wording#

When sharing research, write results as ranges, such as "average trade +0.4% (95% interval minus 0.1% to +0.9%)", rather than a single figure. Readers immediately see how much confidence the evidence supports.

Frequently asked questions#

What is a confidence interval?#

A range of values, calculated from sample data, that is likely to contain the true value of a statistic at a chosen level of confidence, such as 95%.

How do you calculate a 95% confidence interval for a mean?#

Take the sample mean and add and subtract about 1.96 times the standard error.

Why are confidence intervals useful in trading?#

They show how uncertain backtest results are, helping traders avoid trusting statistics from small or noisy samples.

Next, learn to test whether results are real in Hypothesis Testing and P-Values.