# Data Pipelines and ETL > How trading data pipelines extract, transform and load market data reliably. Learn pipeline stages, scheduling, idempotent loads, validation checks and monitoring. Source: https://learn.tradelabsai.com/programming/data-pipelines-and-etl/ Track: Programming and Data · Level: Advanced · Updated: 2026-10-03 Publisher: TradeLabs AI (https://tradelabsai.com). Education, not financial advice. Cite as: TradeLabs Learn, "Data Pipelines and ETL", https://learn.tradelabsai.com/programming/data-pipelines-and-etl/ A data pipeline is the set of steps that moves data from its sources into a form you can use: downloading prices, cleaning them, adjusting for corporate actions and storing them in a database or files. ETL stands for extract, transform and load, the classic three stages. In trading, the pipeline is the foundation under every backtest and live decision. If it silently drops a day, double counts a bar or applies a split twice, every result built on top of it is wrong, and nobody may notice for months. ## The three stages | Stage | What happens | Examples | |---|---|---| | Extract | Get raw data from sources | Vendor files, broker APIs, exchange feeds, websites | | Transform | Clean, standardise and enrich | Fix time zones, remove bad ticks, build bars, compute adjustments | | Load | Store for use | Database tables, Parquet files, caches | A common modern pattern is to store the raw data untouched first, then transform it into clean tables. If a transform has a bug, you can fix it and rebuild from the raw copy. ## A typical daily equity pipeline 1. **After the close,** download the day's bars for every instrument in the universe. 2. **Download corporate actions** such as splits and dividends. 3. **Save the raw files** with the download date. 4. **Validate:** row counts, price ranges, missing symbols, duplicate rows. 5. **Transform:** convert to UTC, map symbols to instrument IDs, compute adjustment factors. 6. **Load** into clean tables. 7. **Report** a summary and alert on any failed check. See [Database Design for Market Data](https://learn.tradelabsai.com/programming/database-design-for-market-data/) and [Splits and Dividends in Price Data](https://learn.tradelabsai.com/programming/adjusted-prices/). ## Idempotent loads A load is idempotent if running it twice gives the same result as running it once. This matters because jobs fail and get rerun. Techniques: - **Unique keys** on (instrument, interval, timestamp) so duplicates are rejected. - **Upserts** that insert new rows or update existing ones. - **Replace by partition:** delete and reload a whole day in one transaction. ## Validation checks | Check | Catches | |---|---| | Row count vs expected | Missing symbols or partial downloads | | High at least open, close and low | Corrupt bars | | Price jump beyond threshold without a corporate action | Bad ticks or missed splits | | Zero or negative prices | Vendor errors | | Volume zero on a normal trading day | Missing data | | Trading calendar comparison | Missing or extra days | See [Cleaning Market Data](https://learn.tradelabsai.com/programming/cleaning-market-data/). **Example: A missed split caught by validation** A pipeline loads a stock that closed at $600 yesterday and opens at $150 today. The jump check flags a 75% drop. The corporate actions table shows a 4 for 1 split effective today, so the move is explained: $600 divided by 4 is $150. The pipeline records the split's adjustment factor of 0.25 for earlier prices and passes. Had the corporate actions download failed, the check would have stopped the load and alerted, instead of letting a backtest believe the stock crashed 75% in a day. ## Scheduling and orchestration Simple pipelines run from cron jobs or scheduled tasks. Larger ones use orchestration tools such as Apache Airflow, Dagster or Prefect, which handle dependencies between steps, retries and alerting. Whatever the tool, every job should log what it did and raise an alert when it fails. See [Monitoring Positions, P&L and Risk](https://learn.tradelabsai.com/algo-trading/live-monitoring/). ## Point in time and versioning Vendors revise data: fundamental figures get restated and prices get corrected. If your pipeline overwrites old values, backtests can use information that was not known at the time. Keeping dated snapshots or recording when each value became known prevents this. See [Point-in-Time and Survivorship-Free Data](https://learn.tradelabsai.com/programming/point-in-time-data/) and [Data Versioning, Lineage and Schemas](https://learn.tradelabsai.com/programming/data-versioning/). ## Streaming pipelines Live trading data flows continuously rather than in daily batches. Streaming pipelines use message queues to pass ticks from feed handlers to bar builders, strategies and storage. See [Message Queues](https://learn.tradelabsai.com/infrastructure/message-queues/) and [Feed Handlers and Normalization](https://learn.tradelabsai.com/programming/feed-handlers-and-normalization/). ## Frequently asked questions ### What is ETL in trading? Extract, transform and load: the process of collecting raw market data, cleaning and standardising it and storing it for research and trading. ### Why do data pipelines need validation? Sources contain errors, gaps and missing corporate actions. Validation catches them before they corrupt backtests and live decisions. ### What does idempotent mean for a data load? Running the load more than once produces the same result as running it once, so reruns after failures do not create duplicates. Next, learn how to fix bad data in [Cleaning Market Data](https://learn.tradelabsai.com/programming/cleaning-market-data/). ## Continue learning - Next lesson: [Cleaning Market Data](https://learn.tradelabsai.com/programming/cleaning-market-data/) - Previous lesson: [Real-Time, Delayed and Historical Data](https://learn.tradelabsai.com/programming/real-time-data/) - Related: [Real-Time, Delayed and Historical Data](https://learn.tradelabsai.com/programming/real-time-data/): What real time market data really means, how it differs from delayed and snapshot data, where it comes from, what it costs and how to judge its quality. - Related: [Cleaning Market Data](https://learn.tradelabsai.com/programming/cleaning-market-data/): Raw market data contains bad ticks, gaps, duplicates and wrong timestamps. Learn how to detect and fix common data errors without distorting your backtests. - Related: [Data Storage, Compression and Caching](https://learn.tradelabsai.com/programming/data-storage/): Compare ways to store market data: CSV, Parquet, HDF5, PostgreSQL, time series and columnar databases. Learn compression, partitioning and how to choose. - Related: [Data Versioning, Lineage and Schemas](https://learn.tradelabsai.com/programming/data-versioning/): Data versioning tracks exactly which data, code and settings produced each backtest. Learn snapshots, hashes, tools like git and DVC, and a simple workflow. - Related: [Database Design for Market Data](https://learn.tradelabsai.com/programming/database-design-for-market-data/): How to design database tables for bars, ticks, symbols, orders and fills. Learn keys, indexes, partitioning, data types and how to avoid common design mistakes. - Related: [Point-in-Time and Survivorship-Free Data](https://learn.tradelabsai.com/programming/point-in-time-data/): Point in time data records what was known on each date, including restated figures and index changes. Learn why it matters and how to build point in time datasets.