Message Queues

As a trading system grows beyond one program, its parts need to talk: feed handlers publish prices, strategies consume them and publish orders, risk services check those orders and loggers record everything. Message queues and messaging systems carry these events between components. They decouple producers from consumers, absorb bursts and allow parts of the system to be restarted or scaled independently. The right choice depends on whether you need speed, durability or both.

Messaging patterns#

Common tools#

Durability versus latency#

Durability costs time: writing to disk and waiting for acknowledgements adds latency. Many systems use a fast path for trading and a separate durable path for recording.

Backpressure#

When consumers fall behind, something must give: messages queue up, get dropped or slow the producer. For market data, the latest price usually matters more than every intermediate one, so conflation (keeping only the newest update per symbol) is common for slow consumers. For orders and fills, nothing may be dropped. Decide this explicitly for each stream.

Ordering and duplicates#

• Ordering: many systems guarantee order only within a partition or topic; route all events for one instrument or account to the same partition.
• Duplicates: at least once delivery means a message can arrive twice; consumers should be idempotent. See Alerts, Error Handling and Reconnection and Sequence Numbers, Dropped Packets and Out-of-Order Messages.

Do you need a message queue?#

A single bot in one process can pass events through an in memory queue with no external system. Introduce messaging when you have several processes, need to record streams durably or want to scale parts independently. See Building Trading Bots.

Frequently asked questions#

What is a message queue in trading?#

A system that passes events such as prices, signals and orders between components, decoupling producers from consumers.

Is Kafka good for trading?#

Kafka is excellent for durable, replayable data pipelines and audit streams, but it is not usually used on the lowest latency trading path.

What is backpressure?#

What happens when consumers cannot keep up with producers; systems must queue, drop, conflate or slow messages, and the right choice depends on the stream.

Next, learn how Redis and PostgreSQL fit into trading setups in Redis and PostgreSQL for Trading.