# Message Queues

> Message queues pass market data, signals and orders between trading components. Learn pub sub and queues, tools like Kafka, Redis and ZeroMQ, and design trade offs.

Source: https://learn.tradelabsai.com/infrastructure/message-queues/  
Track: Trading Infrastructure · Level: Advanced · Updated: 2026-10-03  
Publisher: TradeLabs AI (https://tradelabsai.com). Education, not financial advice.  
Cite as: TradeLabs Learn, "Message Queues", https://learn.tradelabsai.com/infrastructure/message-queues/

As a trading system grows beyond one program, its parts need to talk: feed handlers publish prices, strategies consume them and publish orders, risk services check those orders and loggers record everything. Message queues and messaging systems carry these events between components. They decouple producers from consumers, absorb bursts and allow parts of the system to be restarted or scaled independently. The right choice depends on whether you need speed, durability or both.

## Messaging patterns

| Pattern | How it works | Trading example |
|---|---|---|
| Publish and subscribe | Producers publish to topics; every subscriber gets a copy | Price updates sent to several strategies |
| Work queue | Each message goes to one of several workers | Distributing backtest jobs |
| Request and reply | A request gets a direct response | Asking a risk service to approve an order |
| Event log | Messages stored in order and can be replayed | Recording all events for audit and replay. See [Market Data Replay](https://learn.tradelabsai.com/programming/market-data-replay/) |

## Common tools

| Tool | Character | Typical trading use |
|---|---|---|
| Apache Kafka | Durable, high throughput event log; consumers can replay history | Data pipelines, audit streams, analytics |
| Redis (pub sub and Streams) | In memory, simple, fast | Live prices to dashboards, light messaging. See [Redis and PostgreSQL for Trading](https://learn.tradelabsai.com/infrastructure/redis-and-postgresql-for-trading/) |
| RabbitMQ | Flexible routing, reliable delivery | Job queues, back office workflows |
| ZeroMQ | A library for direct, brokerless messaging with low latency | Fast links between processes |
| NATS | Lightweight, fast pub sub | Service messaging |
| Shared memory ring buffers | Custom, lowest latency | Inside latency critical trading engines. See [Lock-Free Programming and Ring Buffers](https://learn.tradelabsai.com/infrastructure/lock-free-programming/) |

## Durability versus latency

| Need | Choose |
|---|---|
| Every message stored and replayable | Kafka or Redis Streams with persistence |
| Lowest latency between processes on one machine | Shared memory or ZeroMQ over interprocess transport |
| Simple fan out of live prices | Redis pub sub or NATS |
| Reliable job distribution | RabbitMQ or a work queue |

Durability costs time: writing to disk and waiting for acknowledgements adds latency. Many systems use a fast path for trading and a separate durable path for recording.

**Example: A slow consumer**
A feed handler publishes 5,000 price updates per second through Redis pub sub. A dashboard subscriber can process only 2,000 per second. Redis pub sub does not store messages for slow subscribers; the subscriber's output buffer grows until Redis disconnects it at its configured limit. The trading strategy, a separate fast subscriber, is unaffected. The fix for the dashboard is to receive only the latest price per symbol every 100 milliseconds instead of every update. Choosing the right pattern per consumer avoids letting one slow reader cause trouble.

## Backpressure

When consumers fall behind, something must give: messages queue up, get dropped or slow the producer. For market data, the latest price usually matters more than every intermediate one, so conflation (keeping only the newest update per symbol) is common for slow consumers. For orders and fills, nothing may be dropped. Decide this explicitly for each stream.

## Ordering and duplicates

- **Ordering:** many systems guarantee order only within a partition or topic; route all events for one instrument or account to the same partition.
- **Duplicates:** at least once delivery means a message can arrive twice; consumers should be idempotent. See [Alerts, Error Handling and Reconnection](https://learn.tradelabsai.com/algo-trading/error-handling/) and [Sequence Numbers, Dropped Packets and Out-of-Order Messages](https://learn.tradelabsai.com/programming/sequence-numbers/).

## Do you need a message queue?

A single bot in one process can pass events through an in memory queue with no external system. Introduce messaging when you have several processes, need to record streams durably or want to scale parts independently. See [Building Trading Bots](https://learn.tradelabsai.com/programming/building-trading-bots/).

## Frequently asked questions

### What is a message queue in trading?

A system that passes events such as prices, signals and orders between components, decoupling producers from consumers.

### Is Kafka good for trading?

Kafka is excellent for durable, replayable data pipelines and audit streams, but it is not usually used on the lowest latency trading path.

### What is backpressure?

What happens when consumers cannot keep up with producers; systems must queue, drop, conflate or slow messages, and the right choice depends on the stream.

Next, learn how Redis and PostgreSQL fit into trading setups in [Redis and PostgreSQL for Trading](https://learn.tradelabsai.com/infrastructure/redis-and-postgresql-for-trading/).

## Continue learning

- Next lesson: [Redis and PostgreSQL for Trading](https://learn.tradelabsai.com/infrastructure/redis-and-postgresql-for-trading/)
- Previous lesson: [Docker and Kubernetes](https://learn.tradelabsai.com/infrastructure/docker-and-kubernetes/)
- Related: [Docker and Kubernetes](https://learn.tradelabsai.com/infrastructure/docker-and-kubernetes/): How containers with Docker package trading bots and research environments, when Kubernetes helps, and when simpler setups are better for latency and reliability.
- Related: [Feed Handlers and Normalization](https://learn.tradelabsai.com/programming/feed-handlers-and-normalization/): Feed handlers connect to exchange data feeds and convert each venue's format into one standard internal format. Learn the design, symbol mapping and common pitfalls.
- Related: [Redis and PostgreSQL for Trading](https://learn.tradelabsai.com/infrastructure/redis-and-postgresql-for-trading/): How Redis and PostgreSQL work together in trading systems: Redis for live prices, caches and state, PostgreSQL for durable orders, fills and history.
- Related: [Lock-Free Programming and Ring Buffers](https://learn.tradelabsai.com/infrastructure/lock-free-programming/): Lock free data structures let trading threads share data without waiting on locks. Learn ring buffers, atomics, the LMAX Disruptor pattern and the pitfalls involved.
- Related: [Exchange vs Receive Timestamps and Latency Measurement](https://learn.tradelabsai.com/programming/latency-measurement/): How to measure latency in a trading system: where to timestamp, tick to trade and order round trip, percentiles instead of averages and how to find bottlenecks.
- Related: [Building Trading Bots](https://learn.tradelabsai.com/programming/building-trading-bots/): How to build a trading bot that is safe to run: the main components, an event loop, state and position tracking, risk checks, logging and a staged path to live.