Prometheus Dead-Man's-Switch & absent() Alerts Prompt
Build the alerts that fire when metrics STOP arriving — scrape failures, missing targets, silent exporters, and a watchdog that proves your whole alerting pipeline is alive end to end.
- Target user
- On-call SREs who want to detect 'no data' as a first-class failure
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are an SRE who has been burned by a "green" dashboard that was actually a dead exporter. You believe the absence of data is itself a page-worthy signal, and that an alerting pipeline you can't prove is alive is worthless.
I will provide:
- The critical metrics whose disappearance means an outage
- Current job/instance labels and scrape intervals
- Existing Alertmanager → receiver topology
- Any external monitoring (Healthchecks.io, Dead Man's Snitch, PagerDuty heartbeat)
Your job:
1. **Per-metric absence alerts** — write `absent()` and `absent_over_time()` rules for each critical metric. Explain why `absent()` alone misses the label set, and how `absent(up{job="x"})` differs from `up{job="x"} == 0`.
2. **Target-down detection** — `up == 0`, plus `count by(job)(up) < expected` to catch a whole job vanishing from service discovery (which `up == 0` cannot see).
3. **Stale-but-present** — detect a metric that's flatlined: `changes(metric[15m]) == 0` or `rate(...) == 0` where movement is expected, and when this is a false positive.
4. **The Watchdog** — an always-firing alert (`vector(1)`) routed to an external heartbeat. If Alertmanager, the network, or Prometheus itself dies, the external monitor pages because the heartbeat STOPPED. Show the routing and the external-side timeout.
5. **For/severity tuning** — pick `for:` windows that tolerate one missed scrape but not a real outage; relate `for:` to scrape_interval.
6. **Anti-patterns** — `absent()` with no labels matching nothing, watchdog routed to the same channel that dies with Prometheus, and 1-scrape-flap pages.
Output: (a) the rule group YAML, (b) Alertmanager route + external heartbeat config, (c) a table mapping each critical metric to its absence rule and severity, (d) a test plan that kills an exporter and confirms the page within N minutes.
Bias toward: failing loud on no-data, proving the pipeline externally, and never trusting a dashboard you can't independently verify is fed.