# Random Forests and Gradient Boosting

> Random forests and gradient boosted trees are strong models for tabular trading data. Learn how they work, key settings, feature importance and overfitting risks.

Source: https://learn.tradelabsai.com/machine-learning/random-forests/  
Track: Machine Learning · Level: Advanced · Updated: 2026-10-03  
Publisher: TradeLabs AI (https://tradelabsai.com). Education, not financial advice.  
Cite as: TradeLabs Learn, "Random Forests and Gradient Boosting", https://learn.tradelabsai.com/machine-learning/random-forests/

Tree ensemble models are among the most effective machine learning tools for the kind of data traders typically use: tables of features such as returns, volatility, valuation ratios and sentiment scores. A random forest builds many decision trees on random variations of the data and averages them. Gradient boosting builds trees one after another, each correcting the errors of those before it. Both capture non linear relationships and interactions between features without heavy preparation, which makes them popular starting points for quantitative research.

## From one tree to a forest

A single decision tree splits data with simple questions: is the 20 day return above 2%? Is volatility below 15%? It is easy to understand but tends to memorise training data. A random forest reduces this by:

1. **Bootstrapping:** each tree trains on a random sample of rows drawn with replacement.
2. **Feature subsampling:** each split considers only a random subset of features.
3. **Averaging:** predictions from hundreds of trees are averaged or voted.

The randomness makes trees different from each other, and averaging different errors cancels much of the noise.

## Gradient boosting

| | Random forest | Gradient boosting |
|---|---|---|
| Tree building | Independent, in parallel | Sequential, each fixes previous errors |
| Typical tree depth | Deep | Shallow |
| Overfitting risk | Lower with defaults | Higher; needs careful tuning |
| Accuracy on tabular data | Strong | Often the strongest |
| Popular libraries | scikit-learn | XGBoost, LightGBM, CatBoost |

## Key settings

| Setting | Effect |
|---|---|
| Number of trees | More trees reduce variance, with diminishing returns |
| Maximum depth | Deeper trees capture more complexity and overfit more |
| Minimum samples per leaf | Higher values smooth predictions; useful for noisy financial data |
| Features per split | Fewer features add randomness |
| Learning rate (boosting) | Smaller rates need more trees but generalise better |
| Early stopping (boosting) | Stops adding trees when validation error stops improving |

For noisy market data, shallow trees and large minimum leaf sizes usually work better than the defaults designed for cleaner problems.

## Feature importance

Tree models report which features they relied on. Two common methods:

- **Impurity based importance:** how much each feature reduced error in splits. Fast, but biased toward features with many unique values.
- **Permutation importance:** how much performance drops when a feature's values are shuffled. More reliable, especially when measured on out of sample data.

Importance shows what the model used, not that the relationship is real or causal. See [Feature Engineering](https://learn.tradelabsai.com/machine-learning/feature-engineering/).

**Example: Importance that pointed to a leak**
A gradient boosted model predicting next day stock returns shows one feature far more important than all others: "volume change". Its test accuracy is 64%, suspiciously high. Investigation reveals the feature was computed from the full day's volume including the target day itself. After recalculating it with only data known at the prior close, its importance falls to the middle of the pack and accuracy drops to 51%. Feature importance did not prove the model wrong, but an unusually dominant feature is a classic sign of leakage worth checking. See [Data Leakage](https://learn.tradelabsai.com/research/data-leakage/).

## Why trees suit financial features

- **No need to scale** features.
- **Robust to outliers** in inputs, since splits depend on order, not magnitude. See [Outliers and Robust Statistics](https://learn.tradelabsai.com/math/outliers-and-robust-statistics/).
- **Capture interactions,** such as momentum working only in low volatility regimes.
- **Handle mixed feature types.**

## Limitations

- **Cannot extrapolate:** predictions stay within the range of training targets.
- **Can still overfit** noisy data, especially boosting with deep trees.
- **Struggle with raw sequences** compared with specialised models; engineered features work better.
- **Non stationarity** means relationships learned years ago may not hold. See [Structural Breaks and Regime Changes](https://learn.tradelabsai.com/math/regime-changes/).

## Validation

Always validate with time aware methods, such as walk forward or purged cross validation, never random shuffles. See [Model Evaluation and Cross-Validation](https://learn.tradelabsai.com/machine-learning/cross-validation/) and [Walk-Forward Validation and Preventing Overfitting](https://learn.tradelabsai.com/machine-learning/walk-forward-validation/).

## Frequently asked questions

### Are random forests good for trading?

They are a solid, robust choice for tabular features, though their predictions must still be validated carefully and translated into trades with costs.

### What is the difference between random forests and gradient boosting?

Random forests average many independent deep trees; gradient boosting builds shallow trees sequentially, each correcting earlier errors.

### What does feature importance tell me?

Which features the model relied on most. It does not prove a real or causal relationship and can reveal data leakage.

Next, learn how neural networks work and where they fit in [Neural Networks and Deep Learning](https://learn.tradelabsai.com/machine-learning/neural-networks/).

## Continue learning

- Next lesson: [Neural Networks and Deep Learning](https://learn.tradelabsai.com/machine-learning/neural-networks/)
- Previous lesson: [Regression and Classification Models](https://learn.tradelabsai.com/machine-learning/classification-models/)
- Related: [Regression and Classification Models](https://learn.tradelabsai.com/machine-learning/classification-models/): How classification models predict up or down moves and trade outcomes. Learn logistic regression, probability thresholds, precision, recall and confusion matrices.
- Related: [Feature Engineering](https://learn.tradelabsai.com/machine-learning/feature-engineering/): Features are the inputs that give trading models a chance. Learn the main feature families, how to make them stationary and comparable, and how to avoid leakage.
- Related: [Model Evaluation and Cross-Validation](https://learn.tradelabsai.com/machine-learning/cross-validation/): Standard cross validation leaks future data in time series. Learn time series splits, purging and embargo, combinatorial purged cross validation and good practice.
- Related: [Neural Networks and Deep Learning](https://learn.tradelabsai.com/machine-learning/neural-networks/): Neural networks power deep learning, from LSTMs to transformers. Learn how they work, where they help in trading, especially with text and images, and their risks.
- Related: [Overfitting and Curve Fitting](https://learn.tradelabsai.com/research/overfitting-and-curve-fitting/): Overfitting means a strategy fits noise instead of a real pattern. Learn the warning signs, why it happens, how to measure it and practical ways to avoid it.