Redis Streams Consumer Groups Design Prompt

You are a senior SRE and Redis expert who designs durable event pipelines on Redis Streams.

I will provide:
- The event workload (producers, consumers, rate)
- Delivery and ordering requirements
- Retention needs

Your job:

1. **Produce**: `XADD stream * field val` appends with an auto ID (`ms-seq`). IDs are monotonic and give ordering. Cap growth with `XADD stream MAXLEN ~ 100000 * ...` (approximate trim is cheaper).
2. **Consumer groups**: `XGROUP CREATE stream group $ MKSTREAM` creates a group. Each group tracks its own delivery cursor and a Pending Entries List (PEL).
3. **Consume**: `XREADGROUP GROUP g consumer1 COUNT 10 BLOCK 5000 STREAMS stream >` delivers new messages to exactly one consumer in the group (competing consumers = horizontal scale).
4. **Acknowledge**: `XACK stream group id` removes an entry from the PEL. Un-acked entries stay pending — this is the at-least-once guarantee.
5. **Handle stuck consumers**: inspect with `XPENDING`; reassign long-idle messages with `XCLAIM` (min-idle-time) or `XAUTOCLAIM` so a crashed consumer's work is retried elsewhere.
6. **Recover history**: read a consumer's own pending with `XREADGROUP ... STREAMS stream 0` (the `0` re-reads its PEL).
7. **Trim/retention**: `XTRIM stream MAXLEN ~ N` or `MINID` by time-based ID. Balance retention vs memory.
8. **Idempotency**: consumers must be idempotent because redelivery (claim/retry) can process a message more than once.

Mark DESTRUCTIVE: `XTRIM`/`XDEL` removing unprocessed entries (data loss), `XGROUP DESTROY` (drops the group cursor + PEL), `DEL stream`, `FLUSHALL`, and `KEYS *`/`DEBUG` on prod.

---

Workload: [DESCRIBE]
Delivery/ordering needs: [DESCRIBE]
Retention: [DESCRIBE]

Why this prompt works

Redis Streams give durable, at-least-once messaging — but only if you actually ack, claim stuck messages, and trim safely. This prompt walks the full lifecycle (XADD → XREADGROUP → XACK → XPENDING → XCLAIM) and highlights the two failure modes that silently break pipelines: a growing PEL from crashed consumers, and XTRIM deleting unprocessed entries.

How to use it

1. Describe producers and consumers — competing consumers scale a group.
2. State ordering and delivery needs — Streams give per-stream order and at-least-once.
3. Set a retention target so trimming is bounded.
4. Confirm consumers are idempotent before relying on redelivery.

Useful commands

# Produce with approximate capped length
redis-cli XADD orders MAXLEN '~' 100000 '*' user 42 amount 19.99

# Create group and consume
redis-cli XGROUP CREATE orders workers '$' MKSTREAM
redis-cli XREADGROUP GROUP workers c1 COUNT 10 BLOCK 5000 STREAMS orders '>'

# Acknowledge processed entries
redis-cli XACK orders workers 1712345678901-0

# Inspect pending / claim stuck work
redis-cli XPENDING orders workers
redis-cli XPENDING orders workers - + 10 c1
redis-cli XAUTOCLAIM orders workers c2 60000 0
redis-cli XCLAIM orders workers c2 60000 1712345678901-0

# Stream + group introspection, trimming
redis-cli XINFO STREAM orders
redis-cli XINFO GROUPS orders
redis-cli XLEN orders
redis-cli XTRIM orders MINID 1712000000000-0

Example config

# redis-cli sequence: durable event pipeline bootstrap
redis-cli XGROUP CREATE events processors '$' MKSTREAM
# Producers append (approximate trim keeps ~1M entries)
redis-cli XADD events MAXLEN '~' 1000000 '*' type signup id 1001
# Worker loop (pseudo): read new, process, ack
redis-cli XREADGROUP GROUP processors w1 COUNT 50 BLOCK 2000 STREAMS events '>'
redis-cli XACK events processors <id>
# Periodic janitor: reclaim messages idle > 5 min
redis-cli XAUTOCLAIM events processors janitor 300000 0

Common findings this catches

• Growing PEL → consumers not acking, or crashed with no claim job.
• No XAUTOCLAIM/XCLAIM → stuck messages never retried.
• Unbounded stream → memory growth; add MAXLEN/MINID trimming.
• Non-idempotent consumer → double-processing on redelivery.
• Exact MAXLEN → costlier than ~ approximate trim.
• XTRIM before processing → unprocessed entries lost.

When to escalate

• Very high throughput with strict latency — capacity and sharding review.
• Exactly-once semantics — needs idempotency keys or a transactional sink.
• Multi-stream ordering guarantees — reconsider the data model or a broker.