RabbitMQ Publisher Confirms & Idempotent Consumers Prompt
Design an end-to-end no-message-loss path with publisher confirms, durable/persistent messages, and idempotent consumers so neither a broker restart nor a redelivery breaks correctness.
- Target user
- Backend and platform engineers building reliable RabbitMQ pipelines
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior platform engineer who has built RabbitMQ pipelines that survive broker restarts and redeliveries without losing or double-processing messages. Help me make mine bulletproof. I will provide: - The publish and consume flow, and what "losing a message" or "processing twice" would cost [DESCRIBE] - Current durability settings: queue durable flag, message delivery_mode/persistent, queue type [PASTE OUTPUT or DESCRIBE] - Current publisher logic: fire-and-forget, transactions, or confirms [DESCRIBE] - Current consumer ack model: auto-ack or manual ack [DESCRIBE] Your job: 1. **Close the publish gap** — enable publisher confirms (`confirm.select`) so the broker acks each message once it's safe; explain why this beats AMQP transactions on throughput, and handle nacks and timeouts with a resend that doesn't blindly duplicate. 2. **Make messages survive a restart** — durable queue plus persistent messages (`delivery_mode=2`), and note that durable+persistent still has a tiny window unless paired with confirms; recommend quorum queues for the strongest durability where appropriate. 3. **Fix the consume side** — manual acks only after the work is done (never auto-ack for important work), nack/reject with requeue or DLX on failure, and bounded redelivery so a poison message doesn't loop. 4. **Make it idempotent** — because confirms + redelivery give at-least-once, not exactly-once, design consumers to dedupe via a message id / idempotency key so a redelivered message is harmless. 5. **Reason about the whole chain** — walk publisher -> exchange -> queue -> consumer and name every point a message could be lost or duplicated, with the setting that closes each gap. Output as: (a) the end-to-end reliability design, (b) publisher confirm-handling pseudocode, (c) durability settings per queue, (d) the consumer dedupe/idempotency approach. Validate the full path on a staging broker, including a forced broker restart and a forced redelivery, before trusting it in prod. Confirm durability is in place before relying on persistence — a non-durable queue or non-persistent message is lost on restart even with confirms enabled.
Why this prompt works
“No message loss” in RabbitMQ is not a single switch — it’s a chain, and a gap anywhere defeats the rest. The prompt walks the full path from publisher through exchange, queue, and consumer, naming every point a message can be dropped or duplicated and the specific setting that closes it. That end-to-end framing is what stops the common half-measure where someone enables durable queues but still fires publishes blindly, leaving a loss window the durability never covered.
It centers publisher confirms as the right tool for the publish gap, explaining why they beat AMQP transactions on throughput and how to handle nacks and timeouts without blindly duplicating. Crucially, it states plainly that confirms plus redelivery yield at-least-once, not exactly-once delivery — which is exactly why the prompt makes idempotent consumers a first-class requirement rather than an afterthought. A redelivered message has to be harmless, and that means deduping on a message id or idempotency key.
The guardrails target the two beliefs that quietly cause data loss: that confirms alone are enough (they’re not without durable queues and persistent messages, both required), and that redelivery won’t happen (it will, on any reconnect). By demanding a staging test that includes a forced broker restart and a forced redelivery, the prompt makes the team prove the chain holds before betting production correctness on it.