💀 RabbitMQ Dead-Letter Queues and Retry Patterns Done Right With AI

The first dead-letter setup I shipped had an infinite loop baked into it. A poison message would fail, get dead-lettered, get re-queued by my retry logic, fail again, dead-letter again — forever, burning CPU and filling logs. It took me an embarrassingly long time to realize my retry exchange routed messages straight back to the queue they came from with no counter and no ceiling. The topology was valid. It was just wrong in a way RabbitMQ won’t warn you about.

Dead-lettering is where I find AI most useful and most dangerous at the same time. Useful, because the correct retry-with-backoff topology in RabbitMQ involves a non-obvious dance of TTLs, dead-letter exchanges, and a retry counter, and the model can lay all the pieces out at once. Dangerous, because it will confidently produce a setup that loops forever or silently drops messages if you don’t make it justify the dead-letter routing and prove the message actually exits the loop.

Get the model to lay out the full retry topology

The naive question — “how do I add a dead-letter queue?” — gets you a naive answer. Ask for the whole retry-with-backoff pattern and an explicit poison-message exit, because that’s the part people forget.

Design a RabbitMQ retry topology for a payments work queue. I want: failed messages retried with backoff (5s, 30s, 5m), a maximum of 3 retries, and after that the message lands in a payments.parked queue for manual inspection — not re-tried forever. Use dead-letter exchanges and per-message or per-queue TTL. Show me the rabbitmqadmin declarations and explain how a message moves through the topology on each failure, and exactly what stops the loop.

The key thing to grade in the answer: does it track a retry count, and does it have a hard exit to a parked queue? If the design re-queues on failure with no counter, it’s the infinite loop I shipped years ago.

The shape that actually works

The robust pattern uses separate wait queues per backoff tier, each with a TTL that dead-letters back to the work queue when it expires. The retry count lives in a header that the consumer increments.

# Main work queue dead-letters to a retry router on nack
rabbitmqadmin declare exchange name=payments.dlx type=direct durable=true
rabbitmqadmin declare queue name=payments durable=true \
  arguments='{"x-dead-letter-exchange":"payments.dlx","x-dead-letter-routing-key":"retry.5s"}'

# Wait queues — each TTL dead-letters BACK to the work queue
rabbitmqadmin declare queue name=payments.retry.5s durable=true \
  arguments='{"x-message-ttl":5000,"x-dead-letter-exchange":"","x-dead-letter-routing-key":"payments"}'
rabbitmqadmin declare queue name=payments.retry.30s durable=true \
  arguments='{"x-message-ttl":30000,"x-dead-letter-exchange":"","x-dead-letter-routing-key":"payments"}'
rabbitmqadmin declare queue name=payments.retry.5m durable=true \
  arguments='{"x-message-ttl":300000,"x-dead-letter-exchange":"","x-dead-letter-routing-key":"payments"}'

rabbitmqadmin declare binding source=payments.dlx destination=payments.retry.5s routing_key=retry.5s
rabbitmqadmin declare binding source=payments.dlx destination=payments.retry.30s routing_key=retry.30s
rabbitmqadmin declare binding source=payments.dlx destination=payments.retry.5m routing_key=retry.5m

# Terminal parking queue — no TTL, no further dead-lettering
rabbitmqadmin declare queue name=payments.parked durable=true
rabbitmqadmin declare binding source=payments.dlx destination=payments.parked routing_key=parked

The exit logic lives in the consumer: read the x-death header (RabbitMQ stamps it automatically) or your own retry counter, and on the Nth failure, publish to payments.dlx with routing key parked instead of retry.*. AI is good at writing that branch, but you have to ask for it explicitly.

Validate that messages actually leave the loop

This is non-negotiable and it’s the step AI can’t do for you. Inject a message designed to always fail and confirm it lands in payments.parked after exactly three retries — not two, not infinity.

# Seed a poison message
rabbitmqadmin publish exchange="" routing_key=payments \
  payload='{"id":"poison-1","force_fail":true}'

# Watch it bounce through the retry tiers, then check parked
watch -n 5 'rabbitmqadmin list queues name messages | grep payments'

# After it should have given up, confirm it parked exactly once
rabbitmqadmin get queue=payments.parked count=5

Inspect the x-death header on the parked message. It records how many times the message was dead-lettered and by which queue. If the count doesn’t match your retry ceiling, the exit logic is off. I’ve caught off-by-one retry bugs this way that no amount of staring at the topology would have surfaced.

Where AI gets dead-lettering wrong

Two recurring mistakes. First, it sometimes sets the dead-letter routing key such that the message loops in the same tier forever instead of advancing or exiting — always trace the routing keys by hand. Second, it tends to forget that a TTL-on-the-queue dead-letters strictly in order, so a long-TTL message at the head can block shorter-TTL messages behind it (head-of-line blocking). When you mix backoff tiers, give each tier its own queue, which the snippet above does. If the model proposes a single wait queue with per-message TTL, ask it about head-of-line blocking and watch it reconsider.

Also push it on the parked queue: it must have no TTL and no dead-letter exchange, or your “manual inspection” queue will quietly drain itself. Ask: “Confirm the parked queue cannot expire or dead-letter messages.”

The loop I run

Describe the retry-and-park requirements to the AI with the explicit ceiling, get the full topology plus the consumer’s exit branch, apply it to a staging broker, fire a guaranteed-poison message, and verify it parks exactly once after the right number of retries by reading the x-death header. The AI assembles the fiddly multi-queue topology faster than I can and remembers the headers I’d forget. The staging proof that the loop terminates is mine to run.

I keep these retry-topology prompts with the rest of my prompts, and the RabbitMQ category has more on the publisher-confirm and idempotency patterns that make the whole reliability story hold together.