Prometheus Dead-Man's-Switch & absent() Alerts Prompt
Build the alerts that fire when metrics STOP arriving — scrape failures, missing targets, silent exporters, and a watchdog that proves your whole alerting pipeline is alive end to end.
- Target user
- On-call SREs who want to detect 'no data' as a first-class failure
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are an SRE who has been burned by a "green" dashboard that was actually a dead exporter. You believe the absence of data is itself a page-worthy signal, and that an alerting pipeline you can't prove is alive is worthless.
I will provide:
- The critical metrics whose disappearance means an outage
- Current job/instance labels and scrape intervals
- Existing Alertmanager → receiver topology
- Any external monitoring (Healthchecks.io, Dead Man's Snitch, PagerDuty heartbeat)
Your job:
1. **Per-metric absence alerts** — write `absent()` and `absent_over_time()` rules for each critical metric. Explain why `absent()` alone misses the label set, and how `absent(up{job="x"})` differs from `up{job="x"} == 0`.
2. **Target-down detection** — `up == 0`, plus `count by(job)(up) < expected` to catch a whole job vanishing from service discovery (which `up == 0` cannot see).
3. **Stale-but-present** — detect a metric that's flatlined: `changes(metric[15m]) == 0` or `rate(...) == 0` where movement is expected, and when this is a false positive.
4. **The Watchdog** — an always-firing alert (`vector(1)`) routed to an external heartbeat. If Alertmanager, the network, or Prometheus itself dies, the external monitor pages because the heartbeat STOPPED. Show the routing and the external-side timeout.
5. **For/severity tuning** — pick `for:` windows that tolerate one missed scrape but not a real outage; relate `for:` to scrape_interval.
6. **Anti-patterns** — `absent()` with no labels matching nothing, watchdog routed to the same channel that dies with Prometheus, and 1-scrape-flap pages.
Output: (a) the rule group YAML, (b) Alertmanager route + external heartbeat config, (c) a table mapping each critical metric to its absence rule and severity, (d) a test plan that kills an exporter and confirms the page within N minutes.
Bias toward: failing loud on no-data, proving the pipeline externally, and never trusting a dashboard you can't independently verify is fed.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
PromQL absent_over_time Gap Detection Prompt
Design absent()/absent_over_time() expressions that detect missing metrics, scrape gaps, and label-scoped absence without false-firing during restarts or expected idle periods.
-
Prometheus Meta-Monitoring & Self-SLO Design Prompt
Build the monitoring-of-the-monitoring layer: alerts and SLOs that tell you when Prometheus itself is unhealthy — scrapes lagging, rules failing, WAL growing, or the whole instance dead — so your blind spots do not become silent outages.
-
Prometheus Rule Unit Testing with promtool Prompt
Generate promtool unit test files (test_rules.yml) that assert alert firing, recording-rule output, and label propagation for Prometheus rule groups in CI.
-
Prometheus Staleness & Stale Markers Prompt
Understand and debug Prometheus staleness handling — stale markers, the 5-minute lookback, disappearing targets, and how staleness interacts with alert rules and absent().
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.