Point-in-Time and Survivorship-Free Data

A backtest should only use information that was available at each moment in the past. Point in time data makes that possible by recording not just values but when each value became known. This matters because data changes after the fact: companies restate earnings, economic statistics get revised, index members change and vendors correct mistakes. A database that keeps only the latest version silently lets a backtest peek at the future, and the resulting strategies look far better on paper than they could ever have performed.

What changes after the fact#

Two dates for every value#

A point in time record stores:

• Period or effective date: what the value describes, such as the quarter ending 30 June.
• Knowledge date (as of date): when the value was published or entered your database.

A backtest on any day uses only records with a knowledge date on or before that day, and for each item takes the latest such record.

Economic data vintages#

Government statistics are revised as more information arrives. The first estimate of US GDP can differ noticeably from the figure published years later. Markets react to the first release, so a macro backtest should use the initial values, often called first release or real time vintages. The Federal Reserve Bank of St. Louis provides historical vintages through its ALFRED database. See Trading Economic Releases.

Point in time index membership#

Testing a strategy on "today's S&P 500 members" over 20 years only includes companies that survived and grew enough to be in the index today. Real investors in the past would have held companies that later failed or were removed. A point in time membership table stores each stock's add and remove dates so the universe on any day matches what was actually in the index. See Survivorship and Selection Bias.

Building point in time data yourself#

1. Never overwrite; append new versions with a knowledge timestamp. See Data Versioning, Lineage and Schemas.
2. Snapshot vendor data each time you download it.
3. Add a realistic delay where knowledge dates are unclear, for example assume quarterly results are usable 45 to 90 days after quarter end, depending on filing deadlines.
4. Keep delisted instruments in your database.
5. Record your own live data as it arrives, which is point in time by construction.

Commercial point in time datasets#

Major data vendors offer point in time fundamentals, estimates and index constituents, usually at institutional prices. They are expensive because building them requires years of careful archiving. For individuals, the practical approach is adding conservative reporting lags and using survivorship free price data where possible. See Historical Data for Backtesting.

Frequently asked questions#

What is point in time data?#

Data that records when each value became known, so backtests can use only the information that was available on each historical date.

Why do restatements matter for backtesting?#

A backtest using restated figures sees numbers that were not known at the time, creating look ahead bias that inflates results.

How can I avoid look ahead bias with fundamental data?#

Use point in time datasets, or add conservative publication lags so each value is only used after it would have been published.

Next, learn how to track changes to data and code in Data Versioning, Lineage and Schemas.