Sampling and Standard Error

Every statistic you calculate from trading data, such as a win rate, an average return or a Sharpe ratio, is an estimate based on a sample. If you had a different sample of trades or a different period, you would get a different number. The standard error measures how much that estimate would typically vary from sample to sample. It is the key to knowing whether a result is meaningful or just noise, and it explains why small backtests can be so misleading.

Population vs sample#

• Population: all possible outcomes, such as every trade a strategy would ever take.
• Sample: the trades you actually observe.

We use the sample to estimate population values. The standard error tells us how precise that estimate is.

Standard error of the mean#

SE(mean) = s / √n

where s is the sample standard deviation and n is the sample size.

Standard error of a proportion (win rate)#

SE(win rate) = √(p × (1 - p) / n)

Quadrupling the number of trades halves the standard error. See Law of Large Numbers.

Standard error of the Sharpe ratio#

For roughly normal returns, an approximation from Andrew Lo (2002) is:

SE(annual Sharpe) ≈ √((1 + 0.5 × SR²) / years)

A Sharpe ratio of 1.0 measured over 3 years has a standard error of about √(1.5 / 3) ≈ 0.71, so the true Sharpe could plausibly be anywhere from below zero to above 2. Over 10 years, the standard error falls to about 0.39. See Sharpe Ratio.

Sampling problems in trading#

Effective sample size#

If trades or returns are correlated, the effective number of independent observations is smaller than the count. Ten positions in highly correlated stocks taken on the same day are closer to one bet than ten. Using the raw count makes results look more reliable than they are.

Practical guidelines#

1. Always report standard errors or confidence intervals with backtest statistics.
2. Be sceptical of results from fewer than about 30 to 50 independent trades.
3. Prefer longer histories across different market conditions.
4. Adjust for autocorrelation and overlapping data when needed.
5. Use bootstrapping when distributions are unusual. See Bootstrap and Permutation Tests.

A quick sanity check#

Before trusting any backtest figure, divide it by its standard error. A ratio below about 2 means the result could easily be chance. A ratio above 3, from data you did not use to design the strategy, is much more convincing. See Statistical Significance in Trading.

Frequently asked questions#

What is standard error?#

A measure of how much a sample estimate, such as an average or win rate, would vary if you took different samples.

How does sample size affect standard error?#

Standard error shrinks with the square root of the sample size, so four times as many observations halves it.

Why does standard error matter for backtests?#

Because it shows how much of a backtest's result could be due to chance, helping traders avoid trusting lucky results.

Next, learn why averages tend toward a normal shape in Central Limit Theorem.