Failover, Backups and Disaster Recovery

Disaster recovery is the plan for what happens when something big breaks: a server dies, a data centre loses power, the internet goes down, a broker has an outage or a bad deployment corrupts the system. Trading makes these events especially costly because open positions keep moving while you are offline. A good recovery plan defines how quickly you need to be back, how you will get there and how you will manage open risk in the meantime. For a solo trader it can be a one page checklist; for a firm it is a tested, documented process.

Key concepts#

Common disaster scenarios#

Broker side protection#

The single most useful protection for many traders is placing stop orders at the broker rather than only inside a local program. If your system disappears, broker side stops still protect positions. Bracket and OCO orders keep exits alive even when you are offline. See Bracket Orders and OCO Orders. Note that stops do not protect against gaps. See Price Gaps and How to Trade Them.

A recovery runbook#

1. Assess: what failed and what positions and orders are open?
2. Protect: ensure open positions have protective orders; if unsure, reduce risk through any available channel, including the broker's phone desk.
3. Restore: bring up the backup system or restore from backup.
4. Reconcile: confirm positions and orders match the broker's records before trading resumes. See Trade Accounting and Reconciliation.
5. Resume carefully: restart with reduced size or in a monitoring mode.
6. Review: document what happened and improve the plan.

Backups#

A backup that has never been restored is not proven to work. Test restores periodically.

Testing the plan#

Run drills: shut down the main server and fail over, restore a database from backup, practise reaching the broker by phone. Time each step and compare against your recovery time objective. Firms often run scheduled disaster recovery tests, and some regulations require them.

Frequently asked questions#

What is disaster recovery in trading?#

The plan and tools for restoring trading systems and protecting open positions after major failures such as outages, crashes or data loss.

How can retail traders protect against outages?#

Place stops at the broker, keep a mobile app and broker phone number ready, use backup internet and power, and run automated systems on a remote server.

What are RTO and RPO?#

Recovery time objective is how fast you must recover; recovery point objective is how much recent data you can afford to lose.

Next, learn how to keep complete records of every action in Logging, Audit Trails and Incident Response.