RabbitMQ Dead-Letter Exchange & Retry Design Prompt
Design a dead-letter exchange and retry topology so failed messages are retried with backoff and parked safely instead of being lost or stuck in a poison-message loop.
- Target user
- Backend and platform engineers building reliable consumers
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior platform engineer who has designed dead-letter and retry topologies that handle transient failures without losing messages or melting the broker with retry storms. Help me design mine. I will provide: - The consumer flow and what "failure" means (transient downstream error vs permanently malformed message) [DESCRIBE] - Current queue/exchange setup: `rabbitmqadmin list queues` and `list exchanges` with x-arguments [PASTE OUTPUT] - Desired retry behavior: number of attempts, backoff timing, where to park give-ups [DESCRIBE] Your job: 1. **Design the DLX topology** — define the main queue's `x-dead-letter-exchange` (and optional `x-dead-letter-routing-key`), the wait/retry queues, and a final parking queue. Explain how a message dead-letters on nack/reject (with requeue=false), TTL expiry, or max-length overflow. 2. **Implement backoff** — show the per-attempt delay using TTL on dedicated wait queues that dead-letter back to the main queue, or a delayed-message plugin if appropriate; explain the tradeoffs of each approach. 3. **Cap retries** — use a retry counter (from `x-death` header depth or a custom header) to stop after N attempts and route to a parking/dead queue for human inspection, avoiding infinite poison-message loops. 4. **Distinguish failure types** — retry transient failures; route permanently-bad messages straight to parking without burning retries. 5. **Make consumers cooperate** — nack with `requeue=false` to trigger dead-lettering (not `requeue=true`, which loops immediately), and read `x-death` to know attempt count. Output as: (a) a topology diagram (main -> DLX -> wait -> back -> parking), (b) the x-arguments for each queue, (c) consumer nack/ack logic in pseudocode, (d) how to drain and replay the parking queue safely. Validate the DLX topology on a staging broker and prove a message actually retries and parks before relying on it. Confirm the DLX and routing key exist BEFORE setting message TTL or max-length — if the dead-letter target is missing, expired messages are silently discarded.
Why this prompt works
A dead-letter exchange is the difference between a consumer that loses a message the moment something downstream hiccups and one that retries gracefully and parks the truly broken ones for a human. The prompt builds the full topology — main queue, timed wait queues that implement backoff via TTL, and a final parking queue — and ties it to the three real triggers for dead-lettering: a nack/reject with requeue=false, TTL expiry, and max-length overflow. Getting those triggers right is what makes the retry path actually fire.
It addresses the two mistakes that quietly break DLX setups. First, people set a TTL or DLX argument but never confirm the dead-letter target exists, so expired messages vanish instead of being parked. Second, they nack with requeue=true thinking it retries, when it actually requeues to the head of the same queue and creates a tight CPU-burning poison-message loop. The prompt calls both out explicitly and uses the x-death header depth to cap retries.
By separating transient failures (worth retrying with backoff) from permanently malformed messages (send straight to parking), the design avoids wasting retry budget on messages that will never succeed. And by insisting the topology be proven on staging — message actually retries, then parks — it ensures the reliability mechanism is real and not just declared.