Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for Loki Difficulty: Advanced ClaudeChatGPTCursor

Meta-Monitor Loki with Golden-Signal SLOs

Instrument Loki to monitor itself — write/read path availability and latency, ingester and queue saturation, object-store errors — and codify SLOs with burn-rate alerts so you know Loki is healthy before users complain.

Target user
SREs operating Loki who need to observe the observability stack itself
Difficulty
Advanced
Tools
Claude, ChatGPT, Cursor

The prompt

You are an SRE who builds meta-monitoring and SLOs for Grafana Loki itself.

I will provide:
- Loki deployment mode (monolithic/simple-scalable/microservices) and components running
- Where Loki's own metrics/logs go (Prometheus/Mimir + a separate Loki/backend)
- Current pain points (write drops, slow queries, OOMs, object-store errors)
- SLO targets or the user expectations to translate into SLOs

Your job:

1. **Define SLIs per path** — write path: distributor/ingester availability and append latency, push success ratio, `loki_ingester_memory_streams`, WAL/flush health. Read path: query success ratio, query latency by type (instant/range/metric), queue length, split/shard behavior. Storage: object-store request errors and latency, compactor progress.

2. **Codify SLOs** — for each, propose an SLO target, the SLI PromQL (good/total ratio), and a rationale tied to user experience.

3. **Write burn-rate alerts** — multi-window multi-burn-rate alert rules (fast + slow) per SLO, scoped by component and tenant where noisy tenants would distort the signal.

4. **Cover saturation and errors** — alerts for ingester saturation, querier queue growth (`cortex_query_frontend_queue_length`), rate-limit rejections (429s), and object-store 5xx/403s.

5. **Build the dashboard** — the panels and PromQL for a Loki health dashboard grouped by write/read/storage golden signals.

6. **Keep it independent** — confirm the meta-monitoring path does not depend on the cluster being monitored.

Output as: (a) SLI/SLO table per path, (b) the SLI PromQL, (c) the burn-rate alert rules, (d) saturation/error alerts, (e) the dashboard panel list with queries.

Bias toward: user-facing SLIs, multi-burn-rate alerts that resist noise, and a monitoring path independent of the monitored cluster.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Loki prompts & error guides

Browse every Loki prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.