Redis Fork and Copy-on-Write Latency Tuning Prompt
Diagnose and tune Redis fork/copy-on-write latency during RDB and AOF rewrites — THP, overcommit, latest_fork_usec, and COW memory — so background saves stop stalling the primary.
- Target user
- SREs eliminating persistence-induced latency spikes
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior SRE who tunes Redis persistence forking so background saves stop causing latency spikes. I will provide: - `INFO persistence` (rdb_last_fork_usec / latest_fork_usec, aof/rdb in-progress), `INFO stats` - Host RAM, `used_memory`, and OS settings (THP, `vm.overcommit_memory`) - When the spikes occur relative to BGSAVE / AOF rewrite Your job: 1. **Explain the mechanism**: `BGSAVE` and AOF rewrite `fork()` a child that shares the parent's memory copy-on-write. The `fork()` itself pauses the primary while the kernel copies page tables — proportional to dataset size — reported as `latest_fork_usec`. Afterward, every page the primary writes must be COW-duplicated, transiently inflating RSS. 2. **Read the signals**: `INFO persistence` → `latest_fork_usec` (a spike here == the pause), `rdb_last_cow_size`/`aof_last_cow_size` (COW memory used), `rdb_bgsave_in_progress`; correlate with `redis-cli --latency` and app p99 sawtooth aligned to save times. 3. **Kill Transparent Huge Pages**: THP makes COW copy 2 MB pages instead of 4 KB, hugely amplifying fork COW latency and memory. Disabling THP (`madvise`/`never`) is Redis's standard recommendation — Redis even logs a warning when THP is enabled. 4. **Set overcommit**: `vm.overcommit_memory = 1` so `fork()`/`BGSAVE` doesn't fail with "Cannot allocate memory" when free RAM < `used_memory` even though COW means it usually won't be touched. 5. **Reduce fork frequency/cost**: relax `save` points, prefer AOF with `auto-aof-rewrite-percentage` tuned so rewrites are rarer, or offload persistence to a replica (persist there, keep the primary save-light). Smaller datasets fork faster — another reason to shard huge instances. 6. **Ensure headroom**: keep enough free RAM for the COW spike so the fork doesn't trigger the OOM killer mid-save. 7. **Validate**: after changes, confirm `latest_fork_usec` and the p99 spike both drop. Mark DESTRUCTIVE or risky: disabling persistence entirely to hide the spike (data loss), running with no free-RAM headroom (fork OOM), and leaving THP enabled (well-known Redis latency amplifier). --- INFO persistence + host RAM: [PASTE] THP / overcommit settings: [PASTE] Spike timing vs saves: [DESCRIBE]
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Why this prompt works
Persistence-induced latency is one of the most misdiagnosed Redis problems: the primary freezes for tens of milliseconds every time it forks for BGSAVE, and Transparent Huge Pages quietly multiply the pain. This prompt reads latest_fork_usec and the COW sizes directly, applies the three OS fixes Redis itself recommends (THP off, overcommit on, RAM headroom), and reduces fork cost at the source.
How to use it
- Correlate the p99 spikes with save times — if they line up, it’s the fork.
- Check THP and overcommit first — these are the highest-leverage OS knobs.
- Report
latest_fork_usec— it quantifies the pause the whole instance takes.
Useful commands
# The fork pause and COW cost
redis-cli INFO persistence | grep -E 'latest_fork_usec|rdb_last_cow_size|aof_last_cow_size|rdb_bgsave_in_progress|aof_rewrite_in_progress'
# Live latency while a save runs
redis-cli --latency
redis-cli --intrinsic-latency 5
# OS settings Redis cares about (check on the host)
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /proc/sys/vm/overcommit_memory
# Fixes (host-level)
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
sudo sysctl vm.overcommit_memory=1
Example config
# redis.conf — fewer, cheaper forks on a large instance
save 900 1
save 300 100
# rarer AOF rewrites
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 256mb
# (operationally) disable THP and set vm.overcommit_memory=1 on the host
Common findings this catches
- THP enabled → COW copies 2 MB pages; huge fork latency.
overcommit_memory=0→ BGSAVE fails to allocate.- Save-heavy primary → frequent forks; move persistence to a replica.
- No RAM headroom → COW spike triggers OOM killer.
- Huge single dataset → long unavoidable forks; shard it.
When to escalate
- Forks are long purely because the dataset is huge — plan sharding.
- The latency SLO can’t tolerate any fork — persist on a dedicated replica only.
Related prompts
-
Redis Cache Stampede Prevention Design Prompt
Design defenses against cache stampede (dogpile/thundering herd) when a hot Redis key expires and many clients recompute it at once — locks, early recompute, and jittered TTLs.
-
Redis Big Key and Hot Key Analysis Prompt
Hunt down oversized keys and traffic-skewed hot keys that cause latency spikes, uneven cluster load, and blocking deletes — then fix them safely.
-
Redis Multithreaded I/O and Threading Tuning Prompt
Decide whether io-threads, lazy-free background threads, and CPU pinning will actually help — or hurt — a latency-sensitive Redis instance, with real INFO evidence.
-
Redis Persistence RDB/AOF Config Prompt
Configure Redis durability — RDB snapshots vs AOF, appendfsync policy, and hybrid persistence — balancing data safety against latency.
More Redis prompts & error guides
Browse every Redis prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.