Optimization

Optimisation is the process of finding the inputs that produce the best value of some objective: the portfolio weights with the lowest risk for a target return, the model parameters that best fit the data, or the strategy settings with the highest risk adjusted return. Optimisation is everywhere in quantitative trading. It is also dangerous: an optimiser will happily find settings that exploit noise in historical data, producing beautiful backtests that fail live. Knowing how optimisation works, and how to constrain it, is essential.

The parts of an optimisation problem#

Types of problems and methods#

Grid search in strategy design#

Overfitting: the core danger#

The more parameters and combinations you optimise, the more the optimiser fits noise. Signs of overfitting:

• Sharp peaks in performance at specific parameter values, with poor results nearby.
• Big gaps between in sample and out of sample performance.
• Many parameters relative to the number of trades.
• Rules that only make sense in hindsight.

See Overfitting and Curve Fitting.

Making optimisation more robust#

1. Limit free parameters and search ranges.
2. Prefer plateaus over peaks: choose settings where nearby values also work well. See Robustness and Stress Testing.
3. Use walk forward optimisation: optimise on a past window, test on the next, then roll forward. See Walk-Forward Analysis.
4. Hold out data that is never used during development. See In-Sample vs Out-of-Sample Testing.
5. Penalise complexity: use regularisation or information criteria.
6. Adjust for the number of trials when judging results. See Statistical Significance in Trading.

Optimising portfolios#

Mean variance optimisation is notoriously sensitive to inputs. Small changes in expected returns can produce extreme, concentrated weights, an effect Richard Michaud called "error maximisation". Practitioners use constraints, shrinkage of inputs, robust optimisation and simpler methods such as risk parity. See Portfolio Optimization and Risk Budgeting and Risk Parity.

Local vs global optima#

Complex objective functions can have many peaks. Gradient methods may get stuck at a local optimum. Starting from several points, using global methods or simplifying the problem helps. See Calculus for Traders.

Tools#

In Python, SciPy's optimize module handles many problems, cvxpy handles convex problems such as portfolio optimisation, and libraries such as Optuna provide Bayesian optimisation for parameter tuning. See Python for Trading.

Frequently asked questions#

What is optimisation in trading?#

Finding the inputs, such as strategy parameters or portfolio weights, that maximise or minimise an objective like Sharpe ratio or risk.

Why is optimisation dangerous for traders?#

Because it can fit noise in historical data, producing settings that look great in backtests but fail in live trading.

How can I optimise strategies without overfitting?#

Limit parameters, prefer broad stable regions of good results, use walk forward and out of sample testing, and adjust for the number of combinations tried.

Next, learn the statistics of economic data in Econometrics.