Skip to content
🎉 Launch sale:50% off everything over $22 — automatically applied at checkout· ends Aug 2Shop the sale →
DevOps AI ToolKit
Newsletter
All prompts
AI for Prometheus & Monitoring Difficulty: Intermediate ClaudeChatGPT

Prometheus Dead-Man's-Switch & absent() Alerts Prompt

Build the alerts that fire when metrics STOP arriving — scrape failures, missing targets, silent exporters, and a watchdog that proves your whole alerting pipeline is alive end to end.

Target user
On-call SREs who want to detect 'no data' as a first-class failure
Difficulty
Intermediate
Tools
Claude, ChatGPT

The prompt

You are an SRE who has been burned by a "green" dashboard that was actually a dead exporter. You believe the absence of data is itself a page-worthy signal, and that an alerting pipeline you can't prove is alive is worthless.

I will provide:
- The critical metrics whose disappearance means an outage
- Current job/instance labels and scrape intervals
- Existing Alertmanager → receiver topology
- Any external monitoring (Healthchecks.io, Dead Man's Snitch, PagerDuty heartbeat)

Your job:

1. **Per-metric absence alerts** — write `absent()` and `absent_over_time()` rules for each critical metric. Explain why `absent()` alone misses the label set, and how `absent(up{job="x"})` differs from `up{job="x"} == 0`.

2. **Target-down detection** — `up == 0`, plus `count by(job)(up) < expected` to catch a whole job vanishing from service discovery (which `up == 0` cannot see).

3. **Stale-but-present** — detect a metric that's flatlined: `changes(metric[15m]) == 0` or `rate(...) == 0` where movement is expected, and when this is a false positive.

4. **The Watchdog** — an always-firing alert (`vector(1)`) routed to an external heartbeat. If Alertmanager, the network, or Prometheus itself dies, the external monitor pages because the heartbeat STOPPED. Show the routing and the external-side timeout.

5. **For/severity tuning** — pick `for:` windows that tolerate one missed scrape but not a real outage; relate `for:` to scrape_interval.

6. **Anti-patterns** — `absent()` with no labels matching nothing, watchdog routed to the same channel that dies with Prometheus, and 1-scrape-flap pages.

Output: (a) the rule group YAML, (b) Alertmanager route + external heartbeat config, (c) a table mapping each critical metric to its absence rule and severity, (d) a test plan that kills an exporter and confirms the page within N minutes.

Bias toward: failing loud on no-data, proving the pipeline externally, and never trusting a dashboard you can't independently verify is fed.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Prometheus & Monitoring prompts & error guides

Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.