10s Default? SRE Playbook for Prometheus Scrape Timeouts

scrape_timeout is the maximum time Prometheus waits for a target’s /metrics response before giving up. It defaults to 10 seconds and can never exceed scrape_interval, which defaults to 1 minute. The operational fix for most timeout errors is straightforward: measure your target’s p95 response time, set the per-job timeout just above that number, and keep it comfortably under the interval.

---

TL;DR:

• Setting scrape_timeout just above your target’s p95 response time ensures reliable scrapes without unnecessarily delaying failure detection.
• The maximum scrape_timeout is constrained by scrape_interval, which defaults to 1 minute globally, and cannot be set longer than the interval.
• Differentiating between connection/network errors and actual timeout events helps identify whether to increase the timeout or fix the underlying issue.
• Use promtool check config before deploying to verify that your timeout settings do not exceed the interval, avoiding configuration rejections.
• Monitoring scrape_duration_seconds alongside scrape_timeout_seconds enables better troubleshooting, especially when combined with staged testing and load testing of targets.

---

Table of Contents

• What Prometheus Scrape Timeout Actually Controls
• Troubleshooting Prometheus Scrape Timeout Errors
• Editing Prometheus.yml: Timeout Examples and Safe Rollout
• Monitoring Scrape Duration and Related Limits
• Building This Into a Repeatable Playbook
• When Raising the Timeout Is a Fix vs. a Cover-Up
• Automate Your Prometheus Validation Workflow
• Sources

What Prometheus Scrape Timeout Actually Controls

scrape_timeout and scrape_interval sound similar, but they answer two different questions. The interval decides how often Prometheus visits a target. The timeout decides how long it’s willing to wait once it knocks on the door. You can have a slow, infrequent scrape (long interval, long timeout) or a fast, tightly bounded one (short interval, short timeout), but you can’t set a timeout longer than the interval itself.

The official configuration reference sets the global defaults at scrape_interval: 1m and scrape_timeout: 10s, and it states the rule plainly: the timeout “cannot be greater than the scrape interval.” Try to configure it otherwise, and validation will reject the file before Prometheus ever loads it.

Configuration inheritance works top-down. Anything you set under the global scrape_configs block applies to every job, unless that job defines its own scrape_timeout, in which case the per-job value wins. That’s the mechanism you’ll lean on most: leaving the global default alone and overriding timeout on the one or two jobs that actually need more headroom.

A few specifics worth knowing before you touch the config:

• The global default pair is scrape_interval: 1m / scrape_timeout: 10s, and that ratio holds across most managed Prometheus distributions, including Grafana Alloy.
• Per-job scrape_timeout overrides the global value for that job only; everything else still inherits normally.
• promtool check config enforces the timeout-versus-interval constraint at validation time, not at scrape time, so a bad config never reaches production silently.
• When a scrape genuinely times out, Prometheus logs it as a context deadline exceeded error. That phrase is your signal the request was killed for taking too long, not for returning bad data or a connection failure.

That last distinction matters more than it looks. A context deadline exceeded means Prometheus reached the target and started waiting; a connection refused or DNS failure means it never got that far. Confusing the two sends you tuning a timeout when the real problem is a firewall rule or a dead service.

Troubleshooting Prometheus Scrape Timeout Errors

When targets start showing up as down or up == 0 and the logs mention timeouts, work through this in order rather than jumping straight to raising numbers.

1. Read the actual log line. Grep for context deadline exceeded specifically. If you’re instead seeing connection refused, no route to host, or TLS handshake failures, you have a networking or exporter availability problem, not a timeout problem, and no scrape_timeout change will fix it.
2. Query scrape_duration_seconds{job="your_job"} in Prometheus itself and compare it against the configured timeout. If duration is consistently sitting at or near the timeout value, that’s your target confirming it’s actually maxing out the window.
3. Check whether it’s one target or the whole job. A single slow instance points to that host or exporter. Every instance in a job running hot points to something systemic, like a shared backend the exporter queries.
4. Decide: raise the timeout, or fix the exporter. If the target’s own p95 latency is inherently above the current timeout, and that latency is expected (large /metrics payload, expensive collector), raising the per-job timeout is reasonable. If latency is a symptom of an overloaded exporter or an underpowered host, raising the timeout just delays the pain.
5. Validate before you deploy. Run promtool check config prometheus.yml locally so a typo or a timeout-over-interval mistake doesn’t get discovered by an on-call alert.
6. Test in staging first, ideally against a target with production-like payload size and label cardinality, not a lightly loaded dev instance.
7. Rule out adjacent limits. Sample limits, oversized response bodies, and high cardinality can all produce symptoms that look exactly like a timeout, because they slow down how long the exporter takes to serialize its response.

A concrete example from a Stack Overflow thread on this exact behavior: an exporter that takes 3 minutes to respond will simply never appear in Prometheus if scrape_timeout is set to 30 seconds, no matter how generous the scrape_interval is. The interval controls frequency, not patience.

Pro Tip: Before you touch prometheus.yml, curl the exporter’s /metrics endpoint directly and time it under realistic load. If it takes 4 seconds cold and 12 seconds warm with a full cache, your timeout needs to cover the worst case, not the best one.

Editing Prometheus.yml: Timeout Examples and Safe Rollout

Raising a timeout for one noisy job looks like this:

scrape_configs:
  - job_name: 'slow-exporter'
    scrape_interval: 30s
    scrape_timeout: 25s
    static_configs:
      - targets: ['exporter-host:9100']

Note the timeout stays under the interval, with 5 seconds of headroom. If you tried scrape_timeout: 30s on a 30-second interval, promtool would reject it.

If multiple jobs share the same slow characteristics, it’s usually cleaner to raise the global default rather than repeat the override everywhere; this approach helps reduce infrastructure noise impacting agent performance rather than chasing logic bugs.

global:
  scrape_interval: 30s
  scrape_timeout: 20s

Before any of this reaches production:

• Run promtool check config prometheus.yml and read the output carefully. It reports the exact line and job name when a timeout exceeds its interval, so fix the specific job it flags rather than adjusting the whole file blindly.
• Reload with either a SIGHUP to the process or a POST to the /-/reload endpoint if --web.enable-lifecycle is set. Both apply the new config without restarting Prometheus and losing in-memory state.
• Wire promtool check config into your CI pipeline as a pre-merge gate. It’s a single command, and it catches the interval/timeout mismatch long before a bad config ships.
• Watch scrape_duration_seconds for the affected job immediately after reload to confirm the new timeout actually resolved the errors instead of just hiding them longer.

The most common mistake here isn’t a syntax error, it’s setting scrape_timeout equal to scrape_interval instead of comfortably under it. promtool will let you do it in some versions, but it leaves zero margin: any jitter in either the network or the exporter turns into a missed scrape.

Monitoring Scrape Duration and Related Limits

Enable extra_scrape_metrics in your global config, and Prometheus starts exposing scrape_timeout_seconds, scrape_sample_limit, and scrape_body_size_bytes as their own time series, according to the Prometheus configuration docs. Those metrics turn timeout tuning from guesswork into something you can graph and alert on.

A practical alert threshold: fire a warning when scrape_duration_seconds exceeds roughly 80% of the configured timeout for a sustained window, not just once. That gives you a lead indicator before a target actually starts failing scrapes outright.

Watch for signals that mimic a timeout but aren’t:

• A scrape_sample_limit exceeded error, which cuts off ingestion regardless of how fast the target responded.
• A sudden jump in scrape_body_size_bytes, usually caused by a cardinality explosion in one exporter’s labels.
• Several targets in the same job trending toward the timeout ceiling together, which points at a shared dependency rather than one bad host.

There’s a real trade-off buried in all of this. Prometheus’s own self-monitoring metrics, including scrape_duration_seconds, are useful precisely because they distinguish timeout-caused failures from other failure modes. But every second you add to a timeout is also a second longer it takes to notice a target that’s actually dead, not just slow. A generous timeout that masks a truly unresponsive exporter is worse than a strict one that flags it fast. It’s also worth noting that flags like --web.read-timeout and --query.timeout control the Prometheus server’s HTTP and query behavior and are separate from per-scrape timeouts entirely, so don’t confuse a slow dashboard query with a scrape problem.

Building This Into a Repeatable Playbook

Treat timeout changes as a process, not a one-off edit:

• Gate every config change behind promtool check config in CI, before it merges, not after it’s already deployed.
• Load-test the target’s /metrics endpoint in staging under representative traffic to measure real p95 latency before you pick a timeout value.
• Document why a timeout was raised, right next to the config change, and pair it with an alert on scrape_duration_seconds so a regression doesn’t sit unnoticed.
• For scaling issues rather than single-target latency, consider sharding scrape load instead of stretching timeouts across an entire job.
• When the root cause is the exporter itself, tightening how it collects and serializes metrics usually beats permanently loosening the timeout around it.

When Raising the Timeout Is a Fix vs. a Cover-Up

Raising scrape_timeout is the right call when you’ve measured p95 latency, confirmed it’s inherent to the workload, and paired the change with an alert that watches for further drift. It’s the wrong call when it’s a reflex response to a page at 2 a.m.

I’ve seen teams widen a timeout, watch the alert go quiet, and close the ticket without ever asking why the exporter got slow in the first place. That’s not a fix, it’s a snooze button. Use a wider timeout as a bridge while you chase down the actual exporter or network issue, not as the destination. If you can’t point to a specific reason latency increased, don’t raise the number.

— James

Automate Your Prometheus Validation Workflow

Writing safe prometheus.yml diffs, CI gates around promtool check config, and remediation runbooks for timeout incidents takes real time when you’re building it from scratch under pressure. Devopsaitoolkit’s Linux Admin Prompt Pack gives you ready-made diagnostic prompts for exactly this kind of infrastructure troubleshooting, so you’re not drafting a runbook from a blank page while a target is down.

If your CI pipeline needs tighter validation scripting around config checks and reload logic, the Bash Leveled Logging Library prompt helps you build structured logging into those checks so failures are easy to trace later. Pair either with your existing promtool gate, and the next timeout incident comes with a documented history instead of a guess. Grab the prompt pack and have your next scrape timeout runbook drafted before your next on-call rotation starts.

Sources

Start with the Prometheus configuration reference and the command-line flags documentation for the authoritative defaults and validation rules.

• Configuration — Prometheus
• Use of Prometheus scrape_timeout — Stack Overflow

Recommended

• Prometheus Recording Rules That Make Slow Queries Fast
• Scaling Prometheus Scraping: Functional Sharding, Hashmod
• Prometheus Scrape Config and Relabeling Deep Dive
• Blackbox and Synthetic Monitoring With Prometheus