🚨 Prometheus Error Guide: 'HTTP status 429 Too Many Requests' — Scrape Fix

Overview

Prometheus records this on the target status page when the exporter or endpoint being scraped rejects the request with a 429 rate-limit response:

server returned HTTP status 429 Too Many Requests

Some exporters also return a Retry-After header alongside the 429, and cloud/vendor metrics endpoints often do this to protect quota:

server returned HTTP status 429 Too Many Requests (Retry-After: 30)

The scrape is treated as failed: up{} is 0 for that interval and no samples are stored. Unlike a network error, the target is reachable and healthy — it is deliberately shedding load. Fixing it means scraping less aggressively or spreading the load, not retrying harder.

Symptoms

• Target DOWN (or flapping) with server returned HTTP status 429 Too Many Requests.
• up == 0 correlated with short scrape_interval or many Prometheus replicas hitting one endpoint.
• The exporter’s own logs show rate-limit or throttle messages at scrape times.
• Vendor/cloud metrics APIs return 429 as you approach a documented request quota.
• Multiple Prometheus servers (HA pair, plus ad-hoc federation) scrape the same target.

Common Root Causes

• Scrape interval too aggressive for an endpoint with a built-in rate limit.
• Multiple scrapers on one target — an HA Prometheus pair, plus Grafana Agent or a federation pull, all hitting the same exporter.
• Shared multi-target exporter (blackbox/snmp/cloudwatch) overwhelmed by concurrent /probe calls.
• Vendor API quotas — the metrics endpoint is a rate-limited SaaS/cloud API, not a local exporter.
• A misbehaving relabel that duplicated a job, doubling request rate to the same address.
• Exporter concurrency limits set intentionally low, rejecting overlapping scrapes.

Diagnostic Workflow

Confirm the 429 and inspect any Retry-After guidance directly:

curl -sS -D - -o /dev/null http://exporter:9115/metrics | head -n 20

Count how many distinct scrapers reach the target. On the exporter side, look at access logs; from Prometheus, check for duplicate jobs hitting the same address:

count by (instance) (up{job=~".+"}) 

Review the offending job’s cadence and whether HA replicas share the target:

scrape_configs:
  - job_name: cloudwatch
    scrape_interval: 60s        # raise this for rate-limited endpoints
    scrape_timeout: 30s
    static_configs:
      - targets: ["exporter:9106"]

For multi-target exporters, check concurrency and probe volume:

# How many probes are in flight against the shared exporter?
curl -sS http://exporter:9115/metrics | grep -E 'blackbox_module_unknown|probe_'

Watch the failure pattern to confirm it is load-shaped (bursts at scrape boundaries):

changes(up{job="cloudwatch"}[15m])

Example Root Cause Analysis

A cloudwatch_exporter target began returning 429 after a second Prometheus replica was added for HA. Each replica scraped the exporter every 30s, and the exporter in turn called the AWS API — doubling API calls pushed the account past its CloudWatch GetMetricData request quota, and AWS returned 429 which the exporter passed through.

The curl showed 429 with Retry-After: 60. Rather than raising the quota, the fix was to run a single cloudwatch_exporter behind both replicas, increase its cache/scrape interval to 120s, and let each Prometheus scrape the exporter’s cached output instead of triggering fresh API calls. up{} stabilized and API throttling stopped.

Prevention Best Practices

• Match scrape_interval to the endpoint’s rate limit; for vendor APIs, budget scrape frequency against the documented quota.
• Put a caching multi-target exporter in front of rate-limited APIs so many scrapers share one upstream call.
• Avoid pointing multiple independent scrapers at the same expensive exporter; front it with a single exporter and cache.
• Honor Retry-After semantics by choosing an interval comfortably above it.
• Alert on up == 0 with the 429 signature so throttling is distinguished from real outages.

Quick Command Reference

# Confirm the 429 and read Retry-After
curl -sS -D - -o /dev/null http://TARGET:PORT/metrics | head -n 20

up == 0
count by (instance) (up)
changes(up[15m])

Conclusion

A scrape 429 Too Many Requests means the target is healthy but deliberately shedding load. The fix is to scrape less often, deduplicate scrapers, or cache upstream calls behind a single exporter — never to retry more aggressively. Distinguish throttling from outage in your alerts so a rate-limit event doesn’t read as a target being down.