RabbitMQ Consistent Hash Exchange Sharding Prompt
Use the consistent hash exchange to spread load across many queues while keeping all messages for a given key on the same queue, and weight bindings without reshuffling everything.
- Target user
- Platform engineers scaling out ordered or partitioned processing
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior platform engineer who has used the consistent hash exchange plugin to partition RabbitMQ workloads by key. Help me design a sharding scheme that's correct and rebalances cleanly. I will provide: - The partition key (routing key or a header) and why messages for the same key must land on the same queue [DESCRIBE] - The number of shard queues I want and the consumers per shard [DESCRIBE] - Whether shard counts will change over time (scaling up/down) and how much rebalancing I can tolerate [DESCRIBE] - Whether I hash on routing key or a message property/header, and the broker version [DESCRIBE] Your job: 1. **Confirm the model fits** — the consistent hash exchange routes each message to exactly one bound queue chosen by hashing the routing key (or a header). It's for spreading load while keeping per-key affinity, NOT for fan-out. Confirm that's what I need. 2. **Binding weights** — explain that the binding key is a WEIGHT (a number), not a pattern, and that queues with higher weights get proportionally more keys. Recommend equal weights unless I have heterogeneous shard capacity. 3. **Rebalancing on shard change** — describe what happens when I add or remove a shard queue: consistent hashing minimizes but does not eliminate key movement, and in-flight messages already in old queues stay there. Plan a drain strategy for removed shards. 4. **Hash source** — clarify hashing on routing key vs a header (`hash-header` / `hash-property`), and make sure the key has enough cardinality to spread evenly (a low-cardinality key creates hot shards). 5. **Validate** — confirm distribution with `rabbitmqctl list_queues name messages` across shards under load, and verify per-key affinity by tracing a key. Output as: (a) confirmation the consistent hash model fits, (b) the exchange + weighted bindings config, (c) the rebalancing/drain plan, (d) the hash-source choice with a cardinality check, (e) how to verify even spread and key affinity. Validate distribution and rebalancing on staging before production. Test adding and removing a shard and confirm key movement is bounded and old-queue messages drain cleanly — a low-cardinality key produces hot shards that only show up under real traffic.
Why this prompt works
The consistent hash exchange solves a specific, real problem — spreading load across many queues while guaranteeing every message for a given key lands on the same queue — and it gets misused by people who expected fan-out or who treated its binding keys as routing patterns. This prompt opens by confirming the model fits, because if the user actually wants every message in every queue, this is the wrong exchange entirely. Getting that straight first prevents the most expensive class of mistake.
The binding-weight detail is the one that quietly breaks distribution. Unlike every other exchange, the binding key here is a number — a weight — not a pattern, and a queue with a higher weight gets proportionally more keys. Someone copying topic-exchange habits will write a pattern, get nonsense distribution, and not understand why. The prompt makes the weight semantics explicit and defaults to equal weights unless shard capacity genuinely differs.
The two production realities the prompt insists on are rebalancing and key cardinality. Consistent hashing minimizes key movement when shards change but doesn’t eliminate it, and in-flight messages stay in the old queues — so removing a shard needs a drain plan or per-key ordering breaks. And a low-cardinality hash key produces hot shards that no weighting can fix, a failure that only surfaces under real traffic. The staging guardrail makes testing both add and remove of a shard, plus verifying even spread, the precondition for trusting the design.