Hypothesis Testing and P-Values

When a backtest shows profits, the key question is whether the strategy has a real edge or just got lucky. Hypothesis testing is a formal way to answer it. You assume the boring explanation (no edge) and ask how surprising your results would be if that were true. The p value measures that surprise. Hypothesis tests are widely used in trading research, but they are also widely misunderstood and misused, especially when many strategies are tested.

The steps#

1. State the null hypothesis (H0): the default, such as "the strategy's true average return is zero".
2. State the alternative (H1): such as "the true average return is greater than zero".
3. Choose a significance level (α), often 5%.
4. Calculate a test statistic from the data.
5. Find the p value: the probability of results at least this extreme if H0 were true.
6. Decide: if p < α, reject H0; otherwise, do not reject it.

The t test for average returns#

t = (sample mean - hypothesised mean) / (standard deviation / √n)

What a p value is and is not#

The American Statistical Association issued a statement in 2016 warning against these common misinterpretations and against treating p < 0.05 as a magic threshold.

Statistical vs practical significance#

With enough data, tiny effects become statistically significant. A strategy averaging 0.01% per trade over 1 million trades may be "significant" but useless after costs. Always ask whether the effect is large enough to matter. See Statistical Significance in Trading.

Errors#

See Statistical Power and Type I and II Errors.

Multiple testing#

If you test 100 strategies with no edge at α = 5%, about 5 will look significant by chance. Researchers such as Campbell Harvey, Yan Liu and Heqing Zhu (2016) argued that because so many factors have been tested, new trading factors should clear a much higher bar, such as a t statistic above 3 rather than 2. See P-Hacking and Multiple Testing.

Assumptions to check#

• Independence: correlated returns overstate significance. See Autocorrelation and Partial Autocorrelation.
• Distribution: t tests assume roughly normal means; heavy tails can distort results. See Fat Tails.
• Stable process: a changing market undermines tests on historical data. See Structural Breaks and Regime Changes.

When assumptions fail, bootstrap and permutation tests are alternatives. See Bootstrap and Permutation Tests.

Frequently asked questions#

What is a p value?#

The probability of getting results at least as extreme as those observed, assuming the null hypothesis, such as "no edge", is true.

What does p < 0.05 mean?#

That results this extreme would occur less than 5% of the time if there were no real effect; it does not prove the strategy works.

Why can hypothesis tests mislead traders?#

Because testing many strategies produces false positives, assumptions such as independence may fail, and statistical significance does not guarantee practical value.

Next, learn the two ways tests can be wrong in Statistical Power and Type I and II Errors.