In-Sample vs Out-of-Sample Testing

A strategy tuned on historical data will always look good on that same data. The real question is how it performs on data it has never seen. Out of sample testing answers this by setting aside part of the data during development and using it only to evaluate the final strategy. It is one of the most important defences against overfitting and one of the most commonly misused.

In sample vs out of sample#

A common split#

For time series, segments should be in time order: train on earlier data, validate and test on later data. Random splits leak future information into the past. See Data Leakage.

The holdout must stay untouched#

The most common mistake is peeking: testing on the holdout, seeing poor results, adjusting the strategy and testing again. Each time, the holdout becomes part of the development data, and its results become optimistic. If you use the holdout more than once, treat its results as in sample. See P-Hacking and Multiple Testing.

Other forms of out of sample testing#

Limits#

• Short out of sample periods produce noisy estimates. See Sampling and Standard Error.
• Regime differences: the out of sample period may be unusual. See Structural Breaks and Regime Changes.
• Indirect leakage: knowledge of what happened in recent years can influence strategy design even if data is formally held out.
• Multiple strategies: if you test many strategies on the same holdout and pick the best, the holdout is no longer clean.

Best practices#

1. Decide the split before any analysis.
2. Keep the holdout locked until the strategy is final.
3. Use walk forward or several out of sample tests when data allows.
4. Report both in and out of sample results.
5. Expect degradation and judge whether the out of sample result is still worth trading.
6. Follow up with live incubation. See Moving From Paper to Live Trading.

Frequently asked questions#

What is out of sample testing?#

Testing a strategy on data that was not used to develop or tune it, to get an honest estimate of future performance.

How much data should be out of sample?#

Commonly 20% to 30% of the history, in time order, while ensuring both segments are long enough and cover varied market conditions.

Why did my strategy fail out of sample?#

Usually because of overfitting during development, data leakage or a change in market conditions between the two periods.

Next, learn a rolling version of this idea in Walk-Forward Analysis.