Cleaning Market Data: Bad Ticks, Gaps and Errors

Cleaning Market Data

Raw market data contains bad ticks, gaps, duplicates and wrong timestamps. Learn how to detect and fix common data errors without distorting your backtests.

Advanced3 min readUpdated 3 Oct 2026

Lesson 17 of 27

No market data source is perfect. Feeds deliver erroneous trades, vendors miss days, timestamps drift and corporate actions go unrecorded. Backtests are especially sensitive: a single bad tick at one tenth of the real price can create a fictional profitable trade, and a missing week can hide a crash. Cleaning data means finding these problems and fixing or flagging them, while being careful not to remove real but extreme moves, which are often the most important part of the history.

Common data problems#

Problem	Example	Effect on backtests
Bad ticks	A trade printed at $5.00 for a $50 stock	Fake signals and impossible fills
Missing data	No bars for a day the market was open	Hidden losses, wrong indicators
Duplicates	The same bar loaded twice	Double counted volume and returns
Wrong timestamps	Local time mixed with UTC	Misaligned signals. See Timestamps, Time Zones and Daylight Saving
Unadjusted corporate actions	A 2 for 1 split shown as a 50% crash	Fake crashes or rallies. See Splits and Dividends in Price Data
Stale prices	The same close repeated for illiquid days	Understated volatility
Inconsistent bars	High below close	Logic errors

Detecting bad ticks#

Method	How it works
Return threshold	Flag moves larger than a set percentage in one bar
Rolling median filter	Flag prices far from a rolling median
Z score of returns	Flag returns many standard deviations from normal. See Percentiles, Quantiles and Z-Scores
Cross source check	Compare with a second data source
Bid and ask bounds	Flag trades far outside the prevailing quotes

Robust measures such as the median and median absolute deviation work better than the mean and standard deviation, which the bad ticks themselves distort. See Outliers and Robust Statistics.

Real extremes versus errors#

Not every big move is an error. Flash crashes, earnings gaps, the March 2020 crash and crypto liquidation cascades are real. Ways to tell the difference:

Persistence: real moves are followed by trades near the new price.
Volume: real moves usually come with heavy volume.
News or corporate actions explain the move.
Other sources show the same move.
Exchange corrections: venues sometimes cancel clearly erroneous trades, and good data vendors remove them.

Deleting real extremes makes strategies look safer than they are. See Fat Tails.

Handling missing data#

Approach	When to use
Forward fill prices	Short gaps in otherwise liquid data, such as a holiday mismatch
Leave as missing	When the gap is real, such as a trading halt. See Trading Halts and Circuit Breakers
Fill from another source	When the source simply lost data
Drop the instrument	When too much is missing to trust

Never forward fill returns or volume: a filled price implies a zero return, which is reasonable, but filled volume invents trades.

Duplicates and alignment#

Remove exact duplicate rows, and decide rules for near duplicates (same timestamp, different values). When combining assets, align on a trading calendar and check that time zones match. See NumPy and Pandas for Traders.

Document every change#

Keep raw data untouched, apply cleaning rules in code and log what was changed and why. That way results are reproducible and rules can be adjusted later. See Data Versioning, Lineage and Schemas and Backtest Reproducibility.

Frequently asked questions#

What is a bad tick?#

An erroneous price in a data feed, such as a trade recorded far from the real market price, often because of a reporting or entry error.

Should I remove outliers from price data?#

Remove clear errors, but keep real extreme moves, since they represent genuine risk your strategy would have faced.

How do I fill missing price data?#

Forward fill short gaps in prices when appropriate, leave real halts as missing and never invent volume or returns.

Next, learn to handle time correctly in Timestamps, Time Zones and Daylight Saving.

Check your understanding

3 quick questions on this lesson. Get them all right to finish it.

Turn on JavaScript to take the quiz.

Finished this lesson?Sign in to save your progress across devices.

Next lessonTimestamps, Time Zones and Daylight SavingTime zone and timestamp errors silently break backtests. Learn UTC storage, daylight saving traps, exchange sessions, event versus receive time and bar labels.