Data Leakage

Data leakage happens when information that should not be available during model training or decision making slips in, making performance look far better than it will be in reality. Look ahead bias is one form of leakage, but leakage is broader and especially common in machine learning, where complex pipelines can mix training and test data in subtle ways. In trading, leakage turns ordinary models into apparent goldmines, and it is often discovered only after money is lost.

Types of leakage#

Leakage in time series machine learning#

Random k fold cross validation, common in general machine learning, leaks information in financial time series because observations close in time are correlated. Training on Tuesday and testing on Monday of the same week lets the model use information about the future.

Preprocessing leaks#

Target leakage examples in trading#

• Using the day's high or low as a feature to predict whether the day closes up.
• Using end of day volume to predict intraday moves.
• Using analyst upgrades dated to the day when they were actually published after the market close.
• Using an index's constituents that were added because of the very performance being predicted.

How to prevent leakage#

1. Map the timeline: for every feature and label, know exactly when it becomes available.
2. Split data in time order with gaps (purging and embargo) where labels overlap.
3. Build pipelines that fit transformations on training data only.
4. Keep a final holdout untouched. See In-Sample vs Out-of-Sample Testing.
5. Check suspicious results: if a model is "too good", hunt for leaks before celebrating.
6. Test with a deliberate delay: shifting features later in time should hurt performance only modestly.
7. Peer review code and data timing.

Leakage checklist for a new feature#

• When is the raw data published?
• Is it revised later, and which version am I using? See Point-in-Time and Survivorship-Free Data.
• Does any calculation use future rows?
• Could the feature encode the label?
• Are training and test samples independent in time?

Frequently asked questions#

What is data leakage in trading models?#

When information from the future or from the test set slips into model training or features, making performance look unrealistically good.

How is data leakage different from look ahead bias?#

Look ahead bias is a type of leakage involving future information; leakage also includes train test contamination, preprocessing on all data and target leakage.

How do I prevent data leakage in machine learning for trading?#

Split data in time order with purging and embargo gaps, fit preprocessing on training data only, track when each feature becomes available and keep a final holdout.

Next, learn how testing many ideas creates false discoveries in P-Hacking and Multiple Testing.