PromQL absent_over_time Gap Detection Prompt
Design absent()/absent_over_time() expressions that detect missing metrics, scrape gaps, and label-scoped absence without false-firing during restarts or expected idle periods.
- Target user
- SREs and platform engineers running Prometheus building absence and gap alerts
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior observability engineer who builds reliable absence detection in PromQL and knows the subtle differences between absent(), absent_over_time(), and staleness, and how each behaves across restarts and label sets. I will provide: - The metric (and label selectors) that should always be present - The scrape interval and how long a real gap must last before it matters - The known benign gaps (deploys, batch windows, scaling to zero) Your job: 1. **Pick the right function** — choose between `absent()` (instant), `absent_over_time(range)` (sustained absence), and staleness-based detection, and justify the choice for this metric. 2. **Scope the labels correctly** — explain how `absent()` returns the labels you put in the selector, and construct a selector that fires per-instance/per-job as intended rather than one global alert. 3. **Set the window** — size the `absent_over_time` range and the alert `for:` so a single missed scrape or a restart does not page, but a real outage does. 4. **Handle scale-to-zero & batch** — add guards (time-of-day, presence of a parent series) so expected idle periods don't false-fire. 5. **Combine with up** — show when to pair absence detection with `up==0` to distinguish "scrape failed" from "metric genuinely gone". 6. **Write the rule** — produce the alerting rule YAML with labels/annotations and a runbook pointer. 7. **Test it** — give promtool test series that reproduce a real gap and a benign gap to prove the rule fires only on the former. Output as: the chosen expression with rationale, the alerting rule in ```yaml```, and a promtool test snippet covering fire and no-fire cases. Default to caution: prefer a window that tolerates restarts over one that pages on every deploy, and call out explicitly when absence cannot be distinguished from a dropped target without an `up`-based companion alert.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
PromQL Anomaly Detection & Z-Score Alerting Prompt
Build statistical anomaly-detection alerts in pure PromQL — z-score deviation from a rolling baseline, week-over-week seasonal comparison, and MAD-based outlier detection — so you catch weird behavior static thresholds miss.
-
Prometheus Dead-Man's-Switch & absent() Alerts Prompt
Build the alerts that fire when metrics STOP arriving — scrape failures, missing targets, silent exporters, and a watchdog that proves your whole alerting pipeline is alive end to end.
-
Prometheus Alert Rule Generator Prompt
Generate production-quality Prometheus alerting rules with sensible thresholds, labels, and runbook annotations.
-
Prometheus Query Log Slow-Query Audit Prompt
Enable and analyze the Prometheus active query log and query_log_file to find expensive PromQL queries that strain the server, then rewrite or offload them.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.