Prometheus Alert Runbook & Annotation Standardization Prompt
Standardize alert annotations and auto-generate actionable runbooks so every Prometheus alert carries a summary, impact, diagnosis steps, and a remediation link before it ever pages.
- Target user
- SRE leads enforcing alert quality and runbook coverage across teams
- Difficulty
- Beginner
- Tools
- Claude, ChatGPT
The prompt
You are an SRE lead who has cleaned up hundreds of context-free alerts and knows that a good annotation set turns a 3am page from a panic into a checklist.
I will provide:
- A set of existing Prometheus alert rules (the `expr`, current annotations/labels)
- My team's escalation reality (who responds, what tools they have)
- Whether I store runbooks in a wiki, repo, or inline
- Examples of alerts that confused on-call
Your job:
1. **Define the annotation contract** — the required keys every alert must carry: `summary` (one line, no jargon), `description` (what's happening + the firing value), `impact` (user-facing consequence), `runbook_url`, and `dashboard_url`. Explain why each exists.
2. **Template the annotations** — show Go-template annotations that interpolate `{{ $labels.x }}` and `{{ $value }}` so the alert states the actual measured value and the affected entity, not a generic sentence.
3. **Generate a runbook per alert** — for each rule I provide, draft a runbook with: what the alert means, likely causes (ranked), diagnosis commands/queries, remediation steps, and escalation/rollback. Keep it skimmable.
4. **Severity & impact mapping** — map each alert to a severity and a plain-language impact statement; flag any alert whose severity doesn't match its actual user impact.
5. **Link conventions** — define a deterministic `runbook_url` scheme (e.g. `runbooks/<alertname>`) so the link can be generated, not hand-maintained, and dead links are detectable.
6. **CI linter** — give a check (promtool + a small script) that fails a PR when an alert is missing any required annotation or points at a runbook file that doesn't exist.
7. **Backfill plan** — a prioritized order to retrofit existing alerts (start with the ones that page most / confuse most).
8. **Validation** — pick one of my real alerts and show the full before/after: rule annotations + generated runbook + the rendered page text.
Output as: (a) the annotation contract, (b) templated annotation blocks for my rules, (c) one complete runbook per alert, (d) the URL convention, (e) the CI linter script, (f) a before/after example.
Keep it beginner-friendly and prescriptive; give me defaults I can adopt without bikeshedding.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Alert Fatigue Reduction Strategy Prompt
Reduce alert fatigue — SLO-based alerts vs symptom-based, severity tiers, runbook integration, deprecating noisy alerts.
-
Prometheus Meta-Monitoring & Self-SLO Design Prompt
Build the monitoring-of-the-monitoring layer: alerts and SLOs that tell you when Prometheus itself is unhealthy — scrapes lagging, rules failing, WAL growing, or the whole instance dead — so your blind spots do not become silent outages.
-
Prometheus Rule Unit Testing with promtool Prompt
Generate promtool unit test files (test_rules.yml) that assert alert firing, recording-rule output, and label propagation for Prometheus rule groups in CI.
-
PromQL absent_over_time Gap Detection Prompt
Design absent()/absent_over_time() expressions that detect missing metrics, scrape gaps, and label-scoped absence without false-firing during restarts or expected idle periods.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.