Skip to content
🎉 Launch sale:50% off everything over $22 — automatically applied at checkout· ends Aug 2Shop the sale →
DevOps AI ToolKit
Newsletter
All prompts
AI for Prometheus & Monitoring Difficulty: Intermediate ClaudeChatGPT

Prometheus Scrape Timeout & Slow Target Diagnosis Prompt

Diagnose targets that exceed scrape_timeout or return partial data — distinguishing a slow exporter from a slow network from too-large a payload — and fix it without simply raising the timeout until scrapes overlap.

Target user
SREs triaging flaky or slow scrape targets
Difficulty
Intermediate
Tools
Claude, ChatGPT

The prompt

You are a senior observability engineer who can tell from `scrape_duration_seconds` and a curl whether the exporter, the network, or the payload size is to blame.

I will provide:
- The job and target(s) timing out, with their `scrape_interval` and `scrape_timeout`
- Symptoms (intermittent `up=0`, gaps, partial series) and the exporter type
- `scrape_duration_seconds`, `scrape_samples_scraped`, and payload size if I have them

Your job:

1. **Read the built-in scrape metrics** — interpret `scrape_duration_seconds`, `scrape_samples_scraped`, `scrape_samples_post_metric_relabeling`, and `up` to localize the bottleneck.
2. **Isolate exporter vs network** — design a `curl -w` timing test against the target to split DNS/connect/TLS/server-response time from total transfer.
3. **Attack payload size** — identify high-cardinality or oversized exposition responses and reduce them via metric_relabel_configs dropping unneeded series.
4. **Tune timeouts correctly** — explain why scrape_timeout must be < scrape_interval and the failure modes of overlapping scrapes when you raise it too far.
5. **Fix the slow exporter** — common causes (exporter computing metrics on scrape, blocking backends, no caching) and their remedies.
6. **Confirm the fix** — the queries that prove duration dropped and `up` stabilized.

Output as: (a) a decision tree mapping symptom to root cause, (b) the curl timing test and how to read it, (c) the concrete fix for my case, (d) the one thing to never do (e.g. timeout > interval).

Be explicit: raising scrape_timeout is almost always treating a symptom — find why the target is slow before widening the window.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Prometheus & Monitoring prompts & error guides

Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.