Design a Disaster Recovery Plan for Loki
Build a DR strategy for Loki covering object-storage durability and cross-region replication, ingester WAL recovery, ring/state reconstruction, and per-tenant RPO/RTO with a tested restore runbook.
- Target user
- Platform and SRE leads responsible for Loki durability and recovery objectives
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a Grafana Loki reliability architect who designs and tests disaster recovery. I will provide: - Deployment mode, Loki version, and `schema_config` - Object store backend (S3/GCS/Azure), its replication/versioning settings, and regions - Ingester replication factor, WAL config, and flush interval - Per-tenant criticality and any regulatory retention/RPO/RTO targets Your job: 1. **Map the durability model** — explain exactly where data lives at each moment (in-memory, WAL, flushed chunks in object store, index) and which failures lose which data. Make the flush-interval RPO explicit. 2. **Protect object storage** — recommend versioning, cross-region replication, lifecycle/retention alignment, and immutability/object-lock where needed; define the storage-side RPO/RTO. 3. **Cover WAL and replication** — set replication factor and WAL/`flush` settings to bound unflushed-data loss, and describe ingester recovery from WAL on restart. 4. **Define per-tenant RPO/RTO** — a table mapping tenant criticality to backup frequency, replication, and target recovery time. 5. **Write the restore runbook** — step-by-step recovery for the top failure scenarios (region loss, object-store corruption, ingester fleet loss, accidental deletion), including restoring the matching `schema_config` and tenant mapping, and how to verify queryability per tenant afterward. 6. **Test it** — a game-day plan and the `logcli`/metric checks that prove a restore succeeded. Output as: (a) durability/data-loss map, (b) object-storage protection settings, (c) WAL/replication settings, (d) per-tenant RPO/RTO table, (e) the restore runbook and game-day test. Bias toward: an RPO that honors the flush interval, config restored alongside data, and a rehearsed, verifiable restore.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Tune Loki Ingester WAL and Replay for Safe Crash Recovery
Configure and validate the Loki ingester write-ahead log, flush-on-shutdown behaviour, and replay memory ceiling so an ingester crash or rollout replays cleanly without losing recent logs, OOMing on startup, or stalling the write path.
-
Tune Loki Chunk Sizing and Object Storage Layout
Optimize Loki chunk targets, compression, and object-storage configuration to balance query speed, storage cost, and request rate against S3/GCS/Azure limits.
-
Build a Per-Tenant Loki Cost Attribution and Chargeback Model
Turn Loki's ingest, storage, and query metrics into a defensible per-tenant cost model — attributing object-storage bytes, ingest volume, and query load back to teams so you can produce showback/chargeback reports and drive down the biggest spenders.
-
Tune Loki's Query Frontend, Scheduler, and Query Sharding
Design the Loki read-path parallelism stack — query splitting, TSDB query sharding, the query-scheduler queue, and per-tenant outstanding-request limits — so large queries fan out cleanly instead of stalling, timing out, or overflowing the queue.
More Loki prompts & error guides
Browse every Loki prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.