Meta-Monitor Loki with Golden-Signal SLOs
Instrument Loki to monitor itself — write/read path availability and latency, ingester and queue saturation, object-store errors — and codify SLOs with burn-rate alerts so you know Loki is healthy before users complain.
- Target user
- SREs operating Loki who need to observe the observability stack itself
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are an SRE who builds meta-monitoring and SLOs for Grafana Loki itself. I will provide: - Loki deployment mode (monolithic/simple-scalable/microservices) and components running - Where Loki's own metrics/logs go (Prometheus/Mimir + a separate Loki/backend) - Current pain points (write drops, slow queries, OOMs, object-store errors) - SLO targets or the user expectations to translate into SLOs Your job: 1. **Define SLIs per path** — write path: distributor/ingester availability and append latency, push success ratio, `loki_ingester_memory_streams`, WAL/flush health. Read path: query success ratio, query latency by type (instant/range/metric), queue length, split/shard behavior. Storage: object-store request errors and latency, compactor progress. 2. **Codify SLOs** — for each, propose an SLO target, the SLI PromQL (good/total ratio), and a rationale tied to user experience. 3. **Write burn-rate alerts** — multi-window multi-burn-rate alert rules (fast + slow) per SLO, scoped by component and tenant where noisy tenants would distort the signal. 4. **Cover saturation and errors** — alerts for ingester saturation, querier queue growth (`cortex_query_frontend_queue_length`), rate-limit rejections (429s), and object-store 5xx/403s. 5. **Build the dashboard** — the panels and PromQL for a Loki health dashboard grouped by write/read/storage golden signals. 6. **Keep it independent** — confirm the meta-monitoring path does not depend on the cluster being monitored. Output as: (a) SLI/SLO table per path, (b) the SLI PromQL, (c) the burn-rate alert rules, (d) saturation/error alerts, (e) the dashboard panel list with queries. Bias toward: user-facing SLIs, multi-burn-rate alerts that resist noise, and a monitoring path independent of the monitored cluster.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Design Loki Ruler Recording and Alerting Rules
Stand up the Loki ruler to precompute expensive log-metric queries into recording rules and fire alerts on log-derived signals, with correct WAL, remote-write, and per-tenant rule storage.
-
Design Log-Based Alerting with Loki Ruler
Build reliable metric-from-logs alerts using LogQL range aggregations and the Loki ruler, avoiding flaky, high-cardinality, or cost-blowout alert rules.
-
Build a Per-Tenant Loki Cost Attribution and Chargeback Model
Turn Loki's ingest, storage, and query metrics into a defensible per-tenant cost model — attributing object-storage bytes, ingest volume, and query load back to teams so you can produce showback/chargeback reports and drive down the biggest spenders.
-
Tune Loki Ingester WAL and Replay for Safe Crash Recovery
Configure and validate the Loki ingester write-ahead log, flush-on-shutdown behaviour, and replay memory ceiling so an ingester crash or rollout replays cleanly without losing recent logs, OOMing on startup, or stalling the write path.
More Loki prompts & error guides
Browse every Loki prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.