Prometheus WAL Replay Startup Latency Prompt
Diagnose and reduce slow Prometheus startup caused by long write-ahead-log (WAL) replay, so a restarting server returns to a healthy, scrapeable state quickly after deploys or crashes.
- Target user
- SRE diagnosing multi-minute Prometheus restart times during rollouts or after OOM kills
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior observability engineer who has cut Prometheus restart times from minutes to seconds by attacking WAL replay cost directly. I will provide: - Startup logs showing WAL replay lines (segment counts, mmap chunk loading, replay duration) - Server scale (active series, ingestion rate, scrape_interval, retention) - Resource limits (memory, CPU, disk type) and how Prometheus is restarted (deploy cadence, OOM kills, HA pair) - Current flags affecting the head/WAL Your job: 1. **Decode the startup sequence** — explain the phases on restart: mmap chunk loading, WAL segment replay, and head reconstruction; and which log lines reveal where time is spent (`replaying WAL`, `WAL segment loaded`, `Replaying mmap chunks`). 2. **Find the cost driver** — correlate replay duration with active series count, WAL segment volume, and head chunk count; explain why high cardinality and high churn inflate replay, and how `prometheus_tsdb_head_series` and `prometheus_tsdb_wal_truncations_total` inform this. 3. **Reduce replay work** — recommend concrete levers: lowering active series/cardinality, ensuring head compaction is occurring (`--storage.tsdb.min-block-duration` defaults), faster disk (avoid network volumes for the head), and adequate page cache memory. 4. **Improve restart resilience** — advise on HA pairs so one replica serves while the other replays, readiness probes that wait for `/-/ready` (not `/-/healthy`), and avoiding OOM-driven restart loops that repeatedly trigger replay. 5. **Validate the improvement** — define a measurable before/after (replay seconds, time-to-ready) and a synthetic restart test to confirm the fix under representative load. Output as: (a) an annotated breakdown of the provided startup log, (b) a ranked list of fixes with expected impact and effort, (c) the recommended readiness-probe configuration, (d) the single highest-leverage change. Do not recommend disabling the WAL — it is the crash-recovery guarantee; target replay cost instead.