NumPy and Pandas for Traders

NumPy and pandas are the two Python libraries behind almost every trading analysis. NumPy provides fast arrays and maths; pandas builds on it with labelled tables called DataFrames, which are ideal for time series such as prices indexed by date. Once you know a dozen core operations, you can calculate returns, indicators, rolling statistics and simple backtests in a few lines each, running hundreds of times faster than plain Python loops.

The core objects#

Loading and inspecting data#

import pandas as pd
import numpy as np

df = pd.read_csv("btc_1h.csv", parse_dates=["time"], index_col="time")
df = df.sort_index()
print(df.head())
print(df.isna().sum())        # missing values per column
print(df.index.is_unique)     # duplicate timestamps?

Always sort by time and check for missing values and duplicate timestamps before doing anything else. See Cleaning Market Data.

The operations traders use most#

See Rolling and Expanding Windows and Measuring Returns and CAGR.

Resampling bars#

Turning 1 minute bars into 1 hour bars needs a different rule for each column:

hourly = df.resample("1h").agg({
    "Open": "first", "High": "max", "Low": "min",
    "Close": "last", "Volume": "sum",
}).dropna()

Be careful with labels: by default pandas labels each hourly bar by its start time. If your strategy treats the label as the time the bar is complete, you introduce look ahead. See Tick Data and OHLCV Data and Timestamps, Time Zones and Daylight Saving.

Combining several assets#

closes = pd.concat({"SPY": spy["Close"], "TLT": tlt["Close"]}, axis=1)
rets = closes.pct_change().dropna()
print(rets.corr())

concat aligns on dates automatically. Rows where one asset is missing appear as NaN; decide deliberately whether to drop them or fill them. Forward filling a price is usually acceptable for a holiday; forward filling a return is not. See Covariance and Correlation.

Why vectorized code is faster#

A loop in Python processes one number at a time through the interpreter. NumPy and pandas operations hand the whole array to optimised compiled code. On a million rows, a rolling mean in pandas typically finishes in milliseconds while an equivalent Python loop can take seconds. See Event-Driven vs Vectorized Backtesting.

Common traps#

1. Chained assignment such as df[df.a > 0]["b"] = 1, which may silently not change df. Use df.loc[df.a > 0, "b"] = 1.
2. Time zone mixing between naive and aware timestamps.
3. Forgetting shift when turning a signal into a position.
4. Dropping NaNs too early, which misaligns series.
5. Using floats for exact money in accounting code; fine for research, risky for ledgers.

A practice routine#

The fastest way to learn pandas is to answer real questions with it. Download a few years of daily prices for one stock and one index. Then calculate daily returns, the largest one day gain and loss, the average volume by weekday, the 20 day rolling volatility and the correlation between the two assets for each calendar year. Next, resample the daily data into weekly bars and check that the weekly highs equal the highest daily high in each week. Every answer you can verify by hand on a spreadsheet builds trust in your code. Once these feel natural, move on to a full backtest with costs in Backtesting Methodology, and keep each analysis in a script under version control so you can rerun it on new data.

Frequently asked questions#

What is pandas used for in trading?#

Loading, cleaning and transforming price and trade data, calculating returns and indicators, resampling bars and running simple backtests.

Do I need NumPy if I use pandas?#

pandas uses NumPy underneath, and you will use NumPy functions such as np.log and np.sqrt alongside pandas regularly.

How do I calculate returns in pandas?#

Use pct_change() for simple returns or np.log(prices).diff() for log returns.

Next, learn to chart your results in Plotting Market Data with Matplotlib.