Prometheus WAL & TSDB Corruption Recovery Prompt
Diagnose and safely recover a Prometheus instance that fails to start or crash-loops due to WAL replay errors, corrupt blocks, or a full data directory.
- Target user
- SREs and platform engineers running Prometheus responsible for TSDB availability
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior observability engineer who has recovered Prometheus TSDBs after disk-full events, OOM kills mid-compaction, and corrupt WAL segments, and you know which recovery steps lose data versus preserve it. I will provide: - The startup/crash log lines (WAL replay errors, block load errors, or "no space left") - The Prometheus version and storage layout (local disk, PVC, size, retention) - Constraints (can I afford to lose recent data? is this an HA pair?) Your job: 1. **Classify the failure** — distinguish WAL replay corruption, head-block issues, on-disk block (chunk/index) corruption, and disk-full; cite the specific log signature for each. 2. **Snapshot first** — give the exact commands to copy/snapshot the data dir (or PVC) before any mutation, and explain why this is non-negotiable. 3. **Choose the least-destructive path** — order recovery options from safest to most lossy (clear specific WAL segment, drop a single corrupt block, full WAL truncation, last resort wipe), mapping each to the failure class. 4. **Quantify data loss** — for the chosen step, state exactly what time range / which series are lost and whether the HA peer or remote-write/long-term store can backfill it. 5. **Execute** — provide the precise commands (`promtool tsdb`, file removals under `wal/` or `chunks_head/`, block dir deletion) with the service stopped. 6. **Verify & restart** — give the post-recovery checks (`promtool tsdb analyze`, startup log confirmation, `up`/`prometheus_tsdb_head_series`). 7. **Prevent recurrence** — recommend retention/disk headroom, OOM limits, and remote-write so the next failure is non-fatal. Output as: an ordered runbook (numbered steps with copy-pasteable commands), an explicit "data lost" statement for the chosen path, and a prevention checklist. Default to caution: never recommend deleting or truncating anything before a snapshot exists, and if the failure class is ambiguous from the logs, recommend the safest reversible step first.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Prometheus WAL Replay Startup Latency Prompt
Diagnose and reduce slow Prometheus startup caused by long write-ahead-log (WAL) replay, so a restarting server returns to a healthy, scrapeable state quickly after deploys or crashes.
-
Prometheus Native OTLP Receiver Ingestion Prompt
Enable and tune Prometheus's built-in OTLP metrics receiver (/api/v1/otlp/v1/metrics) so OpenTelemetry SDKs and Collectors can push straight into Prometheus with correct name translation, resource-attribute promotion, and out-of-order handling — without a separate remote_write bridge.
-
Prometheus TSDB Head Memory & Series Churn Prompt
Diagnose Prometheus memory pressure driven by the in-memory head block, distinguishing high active-series load from high series churn, and applying the right remediation for each.
-
Prometheus TSDB Snapshot Backup & Restore Prompt
Design a reliable backup and restore procedure for the Prometheus TSDB using the admin snapshot API, object-storage offload, and a tested recovery runbook so you can rebuild a server without silent data loss.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.