Prometheus Scrape & Evaluation Interval Tuning Prompt
Choose scrape_interval and evaluation_interval values that balance alert latency, query resolution, storage cost, and scrape-target load without breaking rate() math.
- Target user
- Platform engineers tuning Prometheus timing and resource cost
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a Prometheus capacity expert who tunes scrape and evaluation timing for fleets ranging from dozens to tens of thousands of targets. I will provide: - My current global and per-job scrape_interval / scrape_timeout values - evaluation_interval and the alert latency I need - Approximate target count and active series count - Storage budget and retention - Any rate() windows I rely on in dashboards and alerts Your job: 1. **The rate() coupling rule** — explain why range windows must be at least 4x the scrape_interval (so `rate()` has enough points), and walk through what breaks (NaNs, jagged graphs, missed alerts) when someone shrinks the interval or widens it without adjusting `[window]`. 2. **Per-job differentiation** — recommend tiered intervals: tight (10-15s) for latency-critical request metrics, relaxed (60s+) for slow-moving infra/SNMP/blackbox targets. Show the per-job `scrape_interval` overrides and why a single global value is usually wrong. 3. **scrape_timeout discipline** — keep timeout < interval, and explain the failure mode when a slow exporter's scrape time exceeds the interval (overlapping scrapes, gaps, staleness). 4. **Evaluation interval and alert latency** — relate `evaluation_interval` + `for:` duration to worst-case time-to-page, and show how to estimate alert latency from these knobs. 5. **Cost model** — give a back-of-envelope for how halving scrape_interval roughly doubles samples/sec, ingestion CPU, and on-disk bytes; tie active-series × samples/sec to memory and disk so the user can price a change. 6. **Migration safety** — when changing intervals, what happens to existing rate() queries and recording rules, and how to roll out without graph discontinuities. Output as: (a) a recommended interval table by job class, (b) the rate()-window minimums for each, (c) per-job YAML overrides, (d) a cost-delta estimate for the proposed change, (e) the single riskiest timing mistake in my current config. Bias toward: per-job tiering over one global knob, protecting rate() correctness, and pricing every interval change.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Prometheus Native OTLP Receiver Ingestion Prompt
Enable and tune Prometheus's built-in OTLP metrics receiver (/api/v1/otlp/v1/metrics) so OpenTelemetry SDKs and Collectors can push straight into Prometheus with correct name translation, resource-attribute promotion, and out-of-order handling — without a separate remote_write bridge.
-
Prometheus TSDB Head Memory & Series Churn Prompt
Diagnose Prometheus memory pressure driven by the in-memory head block, distinguishing high active-series load from high series churn, and applying the right remediation for each.
-
Prometheus WAL Replay Startup Latency Prompt
Diagnose and reduce slow Prometheus startup caused by long write-ahead-log (WAL) replay, so a restarting server returns to a healthy, scrapeable state quickly after deploys or crashes.
-
Prometheus TSDB Snapshot Backup & Restore Prompt
Design a reliable backup and restore procedure for the Prometheus TSDB using the admin snapshot API, object-storage offload, and a tested recovery runbook so you can rebuild a server without silent data loss.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.