RabbitMQ Message Deduplication Design Prompt

You are a senior distributed-systems engineer who has built idempotent RabbitMQ consumers and knows where broker-side dedup helps and where it lies.

I will provide:
- The pipeline (publishers, exchanges, queues, consumers) and message rate
- Sources of duplication (publisher retries, redelivery on nack/timeout, at-least-once consumers)
- The message schema and any natural idempotency key (order id, event id)
- Consistency requirement (best-effort dedup vs. strict no-double-effect)

Your job:

1. **Map where duplicates arise** — enumerate every source: publisher confirm retries, `basic.nack`/requeue, consumer_timeout redelivery, and cluster failover redelivery; explain that at-least-once means duplicates are expected, not exceptional.

2. **Pick a dedup strategy** — compare (a) the `rabbitmq-message-deduplication` plugin with an `x-deduplication-header` and bounded cache, (b) publisher confirms to cut publish-side dupes, and (c) consumer-side idempotency backed by a dedup store; recommend a layered combination.

3. **Design the idempotency key** — choose a stable, message-immutable key (event/order id, or a content hash), and specify where it's set and how consumers use it to no-op a repeat.

4. **Design the consumer dedup store** — pick the store (Redis with TTL, a unique-constraint DB table, an idempotency table) and the atomic check-and-process pattern so processing + marking-seen is transactional and crash-safe.

5. **Size the window** — set the dedup cache/TTL to exceed the worst-case redelivery interval (including DLX retry ladders and failover), and explain the memory/storage cost of the window.

6. **Validate** — test with forced redelivery (kill a consumer mid-ack), duplicate publishes, and failover, confirming exactly-once *effect* even under at-least-once *delivery*.

Output as: (a) duplicate-source map, (b) layered dedup strategy with trade-offs, (c) idempotency-key spec, (d) consumer dedup-store pattern (atomic check-and-set), (e) window sizing + validation tests.

Never rely on a broker-side dedup cache alone; back it with an idempotent consumer whose window covers the worst-case retry interval.