🚨 Prometheus Error Guide: 'target_limit exceeded' — Cap or Filter Discovered Targets

Overview

A scrape job that has target_limit set will refuse to scrape once service discovery hands it more targets than the limit allows. Prometheus logs the rejection and marks the whole scrape pool as failing:

ts=2026-07-06T09:12:44.183Z caller=scrape.go:604 level=error component="scrape manager" scrape_pool=kubernetes-pods msg="target_limit exceeded (number of targets: 512, limit: 200)"

On /api/v1/targets the affected targets carry the same reason in lastError:

"health": "down",
"lastError": "target_limit exceeded (number of targets: 512, limit: 200)",
"scrapePool": "kubernetes-pods"

target_limit is a per-scrape-config cap on the number of active targets (post-relabeling) a job may have. When discovery produces more than the limit, Prometheus rejects the entire target set for that pool rather than scraping a partial subset — so a pool that was healthy can go fully dark after a scale-up.

Symptoms

• A previously green scrape pool goes down all at once after a deployment, node scale-up, or SD change.
• The Prometheus UI Targets page shows the pool with a red target_limit exceeded (number of targets: N, limit: M) error.
• up for the whole job drops to 0; no partial subset keeps reporting.
• Alerts that depend on those series fire no data / absent() even though the exporters are healthy.
• It correlates in time with autoscaling, a new namespace, or a relabeling change that widened what got kept.

Common Root Causes

1. Legitimate growth past a stale cap — the fleet grew (more pods, more EC2 instances) and target_limit was set once and never revisited.
2. Over-broad service discovery — a Kubernetes/EC2/Consul SD that discovers far more than intended because relabel_configs keep too much.
3. A missing or wrong keep/drop relabel rule — a rule regression stops dropping targets that used to be filtered out, doubling or tripling the active set.
4. Label churn creating duplicate targets — relabeling that fails to dedupe (e.g. every container port becomes its own target) inflates the count.
5. A cap copied from a smaller environment — the same job YAML promoted from staging (50 targets) into production (thousands) without resizing the limit.

Diagnostic Workflow

First, confirm the failing pool and the exact numbers from the error:

curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.lastError|test("target_limit")) | [.scrapePool,.health,.lastError] | @tsv' \
  | sort -u

kubernetes-pods  down  target_limit exceeded (number of targets: 512, limit: 200)

Count how many targets discovery is actually producing for that pool (active + dropped) so you can see whether relabeling is filtering as intended:

curl -s 'http://localhost:9090/api/v1/targets?scrapePool=kubernetes-pods' \
  | jq '.data.activeTargets | length, (.data.droppedTargets | length)'

Read the configured limit straight from the running config:

curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' \
  | grep -A15 'job_name: kubernetes-pods' | grep -E 'target_limit|action|regex'

Watch scrape-pool sizing and reloads over time in PromQL:

prometheus_sd_discovered_targets{config="kubernetes-pods"}

count by (job) (up{job="kubernetes-pods"})

Before changing the config, dry-run your relabel rules against real target labels so you can see what should be kept versus dropped:

promtool check config /etc/prometheus/prometheus.yml

Example Root Cause Analysis

A platform team ran kubernetes-pods with target_limit: 200, copied from a staging cluster. After a product launch, the namespace scaled from ~150 to ~500 pods and the whole pool went red with target_limit exceeded (number of targets: 512, limit: 200).

Checking droppedTargets showed only 40 dropped — meaning discovery was keeping almost everything it found. The relabel_configs had a keep rule on __meta_kubernetes_pod_annotation_prometheus_io_scrape=true, but a recent Helm change had added that annotation to a large DaemonSet, so hundreds of extra pods were now legitimately in scope.

Two fixes applied together: (1) the cap was raised to a realistic 600 with headroom over the real count, and (2) a drop rule was added to exclude the DaemonSet’s sidecar port, which had been double-counting. After promtool check config passed and a reload, the active set fell to 480, well under the new limit, and the pool returned to up.

The key lesson: target_limit did its job as a tripwire against runaway discovery, but the durable fix was tightening relabeling, not just raising the number.

Prevention Best Practices

• Set target_limit on every SD-driven job as a tripwire, sized with real headroom (e.g. 1.5–2x current active targets), and revisit it whenever the environment scales.
• Filter at the discovery layer first: use Kubernetes SD selectors, EC2 filters, or tight keep/drop relabel_configs so you don’t discover-then-reject thousands of targets.
• Alert on approach, not just breach: watch prometheus_sd_discovered_targets trending toward the configured limit.
• Keep environment-specific limits — never promote a staging cap into production unchanged.
• Pair target_limit with keep_dropped_targets so a churny SD source can’t also balloon dropped-target memory.
• Review relabel changes in code review specifically for their effect on how many targets get kept.

Quick Command Reference

# Which pools are failing on target_limit, with the numbers
curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.lastError|test("target_limit")) | [.scrapePool,.lastError] | @tsv' | sort -u

# Active vs dropped target counts for a pool
curl -s 'http://localhost:9090/api/v1/targets?scrapePool=kubernetes-pods' \
  | jq '{active: (.data.activeTargets|length), dropped: (.data.droppedTargets|length)}'

# Configured target_limit + relabel rules for the job
curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' \
  | grep -A15 'job_name: kubernetes-pods'

# Validate config and reload after editing
promtool check config /etc/prometheus/prometheus.yml \
  && curl -s -XPOST http://localhost:9090/-/reload

# Discovered-target trend
# (PromQL) prometheus_sd_discovered_targets{config="kubernetes-pods"}

Conclusion

target_limit exceeded is Prometheus protecting a server from a scrape pool that suddenly discovered far more targets than expected. Treat it as a signal, not just a number to raise: confirm the real active-target count, decide whether the growth is legitimate, and prefer tightening service discovery and relabeling over blindly bumping the cap. Right-size the limit with headroom, filter at the SD layer, and pair it with keep_dropped_targets so both scraped and dropped targets stay bounded. Done that way, the same guardrail that took the pool down becomes the early warning that keeps your TSDB and memory footprint predictable as the fleet grows.