Redis HyperLogLog Cardinality Design Prompt
Design HyperLogLog counting on Redis — PFADD/PFCOUNT/PFMERGE, error bounds, key rollups, and Cluster merges — for unique-count analytics.
- Target user
- Senior engineers building unique-count analytics on Redis
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior SRE and Redis data-modeling expert who designs approximate unique-count ("cardinality") analytics on Redis in production.
I will provide:
- What is being counted (unique users, IPs, devices) and the time grain (per minute/hour/day)
- Expected cardinality per bucket and number of buckets
- Read patterns (single bucket, rolling window, merged totals)
- Topology (standalone / Sentinel / Cluster) and accuracy tolerance
Your job:
1. **Confirm HyperLogLog fits.** HLL answers "how many *distinct* items?" with ~0.81% standard error using a fixed **~12 KB** per key regardless of cardinality (up to billions). It cannot list members, test membership, or give exact counts. If the user needs any of those, HLL is the wrong tool — say so and suggest a set or Bloom filter.
2. **Design the key grain.** One HLL key per (metric, time-bucket): e.g. `uv:2026-07-08:homepage`. Smaller grains (per-minute) cost more keys but enable flexible rollups; larger grains (per-day) cost fewer keys but less resolution. Recommend a grain and a key-naming scheme.
3. **Plan rollups with PFMERGE.** Daily = `PFMERGE` of 24 hourly keys; weekly = merge of 7 daily. Merges are unioned correctly (no double counting), which set-based counting cannot match cheaply. Note `PFCOUNT key1 key2 ...` gives the union count without materializing a merge key.
4. **Handle Redis Cluster.** `PFMERGE` and multi-key `PFCOUNT` require all source and destination keys in the **same slot** — otherwise CROSSSLOT. Use a **hash tag** so a rollup family co-locates, e.g. `uv:{homepage}:2026-07-08:14`. Design the tag around the merge boundary.
5. **Set expiry.** Add `EXPIRE`/`EXPIREAT` per bucket so old fine-grained keys drop after they have been rolled up. Keep long-lived aggregate keys only.
6. **Understand the error.** 0.81% relative standard error means a true 1,000,000 might read 990k–1,010k. For dashboards that is fine; for billing it is not. State the tolerance explicitly.
7. **Write path.** `PFADD` is O(1) amortized and idempotent for repeated items — safe to call on every event. It returns 1 only when the estimate likely changed, which you can ignore.
Deliverables: key schema with hash-tag strategy, `PFADD`/`PFCOUNT`/`PFMERGE` templates, rollup + expiry plan, a memory estimate (keys × ~12 KB), and the accuracy statement.
Mark DESTRUCTIVE: `PFMERGE` that overwrites an existing aggregate key, `FLUSHDB`, and replacing an HLL key with a `SET` "for exactness" (blows up memory from 12 KB to unbounded).
---
What/grain: [DESCRIBE]
Cardinality/buckets: [DESCRIBE]
Topology/tolerance: [DESCRIBE]
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Why this prompt works
The whole appeal of HyperLogLog is counting billions of distinct items in a fixed ~12 KB — but that only holds if the key grain, the PFMERGE rollup plan, and the Cluster hash-tag layout are designed together. Teams that skip this end up with CROSSSLOT errors on merges, thousands of un-expired buckets, or someone quietly swapping in a giant SET for “accuracy.” This prompt pins down the grain, the merge boundary, the slot strategy, and the error budget before any code ships.
How to use it
- Name the metric and time grain so the key scheme is concrete.
- Estimate cardinality and bucket count for a memory number.
- Describe the rollups you need (hourly→daily→weekly) to design
PFMERGE. - State the accuracy tolerance — this decides whether HLL is even valid.
Useful commands
# Count uniques into an hourly bucket (idempotent per item)
redis-cli PFADD uv:{homepage}:2026-07-08:14 user:42 user:99
# Estimate distinct count for one bucket
redis-cli PFCOUNT uv:{homepage}:2026-07-08:14
# Union count across hours WITHOUT materializing a key
redis-cli PFCOUNT uv:{homepage}:2026-07-08:13 uv:{homepage}:2026-07-08:14
# Roll 24 hourly keys into a daily aggregate (same slot via hash tag)
redis-cli PFMERGE uv:{homepage}:2026-07-08 \
uv:{homepage}:2026-07-08:00 uv:{homepage}:2026-07-08:01 # ...through :23
# Expire fine-grained buckets after rollup
redis-cli EXPIRE uv:{homepage}:2026-07-08:14 172800 # 48h
# Inspect: an HLL is a special string, ~12 KB dense
redis-cli MEMORY USAGE uv:{homepage}:2026-07-08
redis-cli TYPE uv:{homepage}:2026-07-08 # string
Cluster note
# WRONG: keys land in different slots -> CROSSSLOT
PFMERGE uv:day uv:hour:00 uv:hour:01
# RIGHT: hash tag {homepage} forces one slot for the whole family
PFMERGE uv:{homepage}:day uv:{homepage}:00 uv:{homepage}:01
Common findings this catches
- Exact-count requirement → HLL is invalid; use a set or exact counter.
- CROSSSLOT on PFMERGE → missing hash tag around the rollup family.
- Thousands of un-expired buckets → add TTLs, keep only aggregates.
SET-based unique counting → gigabytes where 12 KB would do.- Membership/listing needed → wrong primitive; HLL can’t do it.
When to escalate
- Sub-1% error is unacceptable — move to exact structures and accept the memory cost.
- Very high bucket counts strain key space — consolidate grain or shard.
- Need both count and membership — pair an HLL with a Bloom filter or set.
Related prompts
-
Redis Data Structure Selection Prompt
Choose the right Redis type — string, hash, list, set, sorted set, stream, bitmap, or HyperLogLog — for a given use case and access pattern.
-
Redis Memory Optimization Prompt
Analyze Redis memory usage — encodings, big keys, fragmentation — and reduce footprint with listpack/intset thresholds and smarter modeling.
-
Redis Cluster Sharding Design Prompt
Design Redis Cluster sharding — 16384 hash slots, resharding, hash tags, and multi-key operation constraints across shards.
-
Redis TTL and Expiration Strategy Prompt
Design TTL hygiene with EXPIRE/PEXPIRE, understand active vs lazy expiry, and avoid immortal keys and expiry-driven latency spikes.
More Redis prompts & error guides
Browse every Redis prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.