Prometheus Scrape Timeout & Slow Target Diagnosis Prompt
Diagnose targets that exceed scrape_timeout or return partial data — distinguishing a slow exporter from a slow network from too-large a payload — and fix it without simply raising the timeout until scrapes overlap.
- Target user
- SREs triaging flaky or slow scrape targets
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior observability engineer who can tell from `scrape_duration_seconds` and a curl whether the exporter, the network, or the payload size is to blame. I will provide: - The job and target(s) timing out, with their `scrape_interval` and `scrape_timeout` - Symptoms (intermittent `up=0`, gaps, partial series) and the exporter type - `scrape_duration_seconds`, `scrape_samples_scraped`, and payload size if I have them Your job: 1. **Read the built-in scrape metrics** — interpret `scrape_duration_seconds`, `scrape_samples_scraped`, `scrape_samples_post_metric_relabeling`, and `up` to localize the bottleneck. 2. **Isolate exporter vs network** — design a `curl -w` timing test against the target to split DNS/connect/TLS/server-response time from total transfer. 3. **Attack payload size** — identify high-cardinality or oversized exposition responses and reduce them via metric_relabel_configs dropping unneeded series. 4. **Tune timeouts correctly** — explain why scrape_timeout must be < scrape_interval and the failure modes of overlapping scrapes when you raise it too far. 5. **Fix the slow exporter** — common causes (exporter computing metrics on scrape, blocking backends, no caching) and their remedies. 6. **Confirm the fix** — the queries that prove duration dropped and `up` stabilized. Output as: (a) a decision tree mapping symptom to root cause, (b) the curl timing test and how to read it, (c) the concrete fix for my case, (d) the one thing to never do (e.g. timeout > interval). Be explicit: raising scrape_timeout is almost always treating a symptom — find why the target is slow before widening the window.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Prometheus Target-Down & Scrape Failure Triage Prompt
Systematically triage why a Prometheus target shows up==0 or scrape errors — distinguishing network, TLS, auth, relabel-drop, and sample-limit causes from the target's scrape metadata.
-
Prometheus Exporter TLS & Auth Hardening Prompt
Secure exporter and scrape endpoints with TLS and authentication using Prometheus web-config and exporter web.config.file so metrics endpoints exposing internal labels and topology are no longer open on the network.
-
Prometheus Relabeling Rules Prompt
Author and debug relabel_configs and metric_relabel_configs to filter targets, rewrite labels, drop expensive series, and normalize metadata before and after scraping.
-
snmp_exporter dot1qTpFdbPort & Network Metrics
Configure the Prometheus snmp_exporter for switches, routers, firewalls and UPSes — generator.yml modules, MIB walks (incl. BRIDGE-MIB dot1qTpFdbPort), v2c/v3 auth, and clean OID-to-label mapping.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.