# Cleaning Market Data

> Raw market data contains bad ticks, gaps, duplicates and wrong timestamps. Learn how to detect and fix common data errors without distorting your backtests.

Source: https://learn.tradelabsai.com/programming/cleaning-market-data/  
Track: Programming and Data · Level: Advanced · Updated: 2026-10-03  
Publisher: TradeLabs AI (https://tradelabsai.com). Education, not financial advice.  
Cite as: TradeLabs Learn, "Cleaning Market Data", https://learn.tradelabsai.com/programming/cleaning-market-data/

No market data source is perfect. Feeds deliver erroneous trades, vendors miss days, timestamps drift and corporate actions go unrecorded. Backtests are especially sensitive: a single bad tick at one tenth of the real price can create a fictional profitable trade, and a missing week can hide a crash. Cleaning data means finding these problems and fixing or flagging them, while being careful not to remove real but extreme moves, which are often the most important part of the history.

## Common data problems

| Problem | Example | Effect on backtests |
|---|---|---|
| Bad ticks | A trade printed at $5.00 for a $50 stock | Fake signals and impossible fills |
| Missing data | No bars for a day the market was open | Hidden losses, wrong indicators |
| Duplicates | The same bar loaded twice | Double counted volume and returns |
| Wrong timestamps | Local time mixed with UTC | Misaligned signals. See [Timestamps, Time Zones and Daylight Saving](https://learn.tradelabsai.com/programming/timestamps-and-time-zones/) |
| Unadjusted corporate actions | A 2 for 1 split shown as a 50% crash | Fake crashes or rallies. See [Splits and Dividends in Price Data](https://learn.tradelabsai.com/programming/adjusted-prices/) |
| Stale prices | The same close repeated for illiquid days | Understated volatility |
| Inconsistent bars | High below close | Logic errors |

## Detecting bad ticks

| Method | How it works |
|---|---|
| Return threshold | Flag moves larger than a set percentage in one bar |
| Rolling median filter | Flag prices far from a rolling median |
| Z score of returns | Flag returns many standard deviations from normal. See [Percentiles, Quantiles and Z-Scores](https://learn.tradelabsai.com/math/z-scores/) |
| Cross source check | Compare with a second data source |
| Bid and ask bounds | Flag trades far outside the prevailing quotes |

Robust measures such as the median and median absolute deviation work better than the mean and standard deviation, which the bad ticks themselves distort. See [Outliers and Robust Statistics](https://learn.tradelabsai.com/math/outliers-and-robust-statistics/).

**Example: Filtering a bad print**
A stock trades around $80. One tick shows $8.00 for 100 shares, followed by more trades at $80.05. The rolling median of the last 50 trades is $80.02, and the median absolute deviation is $0.03. The $8.00 print sits about 2,400 deviations away, while the threshold is 10. The filter marks it as bad. A real gap down, for example after earnings, would show many consecutive trades at the new level and would not be filtered. A backtest without this filter might have "bought" at $8.00 and recorded a 900% gain.

## Real extremes versus errors

Not every big move is an error. Flash crashes, earnings gaps, the March 2020 crash and crypto liquidation cascades are real. Ways to tell the difference:

- **Persistence:** real moves are followed by trades near the new price.
- **Volume:** real moves usually come with heavy volume.
- **News or corporate actions** explain the move.
- **Other sources** show the same move.
- **Exchange corrections:** venues sometimes cancel clearly erroneous trades, and good data vendors remove them.

Deleting real extremes makes strategies look safer than they are. See [Fat Tails](https://learn.tradelabsai.com/math/fat-tails/).

## Handling missing data

| Approach | When to use |
|---|---|
| Forward fill prices | Short gaps in otherwise liquid data, such as a holiday mismatch |
| Leave as missing | When the gap is real, such as a trading halt. See [Trading Halts and Circuit Breakers](https://learn.tradelabsai.com/markets/trading-halts/) |
| Fill from another source | When the source simply lost data |
| Drop the instrument | When too much is missing to trust |

Never forward fill returns or volume: a filled price implies a zero return, which is reasonable, but filled volume invents trades.

## Duplicates and alignment

Remove exact duplicate rows, and decide rules for near duplicates (same timestamp, different values). When combining assets, align on a trading calendar and check that time zones match. See [NumPy and Pandas for Traders](https://learn.tradelabsai.com/programming/numpy-and-pandas-for-traders/).

## Document every change

Keep raw data untouched, apply cleaning rules in code and log what was changed and why. That way results are reproducible and rules can be adjusted later. See [Data Versioning, Lineage and Schemas](https://learn.tradelabsai.com/programming/data-versioning/) and [Backtest Reproducibility](https://learn.tradelabsai.com/research/backtest-reproducibility/).

## Frequently asked questions

### What is a bad tick?

An erroneous price in a data feed, such as a trade recorded far from the real market price, often because of a reporting or entry error.

### Should I remove outliers from price data?

Remove clear errors, but keep real extreme moves, since they represent genuine risk your strategy would have faced.

### How do I fill missing price data?

Forward fill short gaps in prices when appropriate, leave real halts as missing and never invent volume or returns.

Next, learn to handle time correctly in [Timestamps, Time Zones and Daylight Saving](https://learn.tradelabsai.com/programming/timestamps-and-time-zones/).

## Continue learning

- Next lesson: [Timestamps, Time Zones and Daylight Saving](https://learn.tradelabsai.com/programming/timestamps-and-time-zones/)
- Previous lesson: [Data Pipelines and ETL](https://learn.tradelabsai.com/programming/data-pipelines-and-etl/)
- Related: [Data Pipelines and ETL](https://learn.tradelabsai.com/programming/data-pipelines-and-etl/): How trading data pipelines extract, transform and load market data reliably. Learn pipeline stages, scheduling, idempotent loads, validation checks and monitoring.
- Related: [Outliers and Robust Statistics](https://learn.tradelabsai.com/math/outliers-and-robust-statistics/): Outliers can distort averages, correlations and backtests. Learn how to detect them, robust measures like the median and MAD, winsorising and when they matter.
- Related: [Splits and Dividends in Price Data](https://learn.tradelabsai.com/programming/adjusted-prices/): Adjusted prices remove the jumps caused by splits and dividends so returns are correct. Learn how adjustment factors work, when to use raw prices and common traps.
- Related: [Timestamps, Time Zones and Daylight Saving](https://learn.tradelabsai.com/programming/timestamps-and-time-zones/): Time zone and timestamp errors silently break backtests. Learn UTC storage, daylight saving traps, exchange sessions, event versus receive time and bar labels.
- Related: [Historical Data for Backtesting](https://learn.tradelabsai.com/research/historical-data-for-backtesting/): Backtests are only as good as their data. Learn data types and sources, quality checks, corporate action adjustments and how to avoid survivorship traps.