🚨 Prometheus Error Guide: 'server returned HTTP status 503 Service Unavailable' — Fix Failing Scrapes

Overview

Prometheus marks a target as down and records this scrape error whenever the target’s /metrics endpoint answers with an HTTP 503. It appears in the target’s Last Error on the /targets page and in the server log:

level=debug component="scrape manager" scrape_pool=api target=http://10.0.4.11:8080/metrics
  msg="Scrape failed" err="server returned HTTP status 503 Service Unavailable"

Unlike a connection refused or timeout, the TCP connection and HTTP exchange succeeded — the target (or something in front of it) deliberately returned 503. The scrape is discarded, up{...} for that target flips to 0, and every series from that target goes stale.

Symptoms

• up{job="..."} == 0 for one or many targets while the process itself is running.
• /targets shows state DOWN with server returned HTTP status 503 Service Unavailable.
• The 503 is intermittent (target under load) or constant (target not ready, or a proxy short-circuiting).
• Other endpoints of the same service may work while /metrics specifically 503s.
• scrape_samples_scraped drops to nothing for the affected targets.

Common Root Causes

• Target still starting up — the app serves 503 from a readiness/health layer until initialization completes, so early scrapes fail.
• Overloaded application — the metrics handler shares a worker pool with request traffic and sheds load with 503 under saturation.
• Ingress/proxy in the path — an nginx, Envoy, or cloud load balancer returns 503 (no healthy upstream) rather than the app; Prometheus is scraping the proxy, not the pod.
• Kubernetes endpoint churn — the Service has no ready endpoints during a rollout, so the LB answers 503.
• Rate limiting / maintenance mode — the app returns 503 with Retry-After when throttling or draining.
• Metrics endpoint gated behind auth that returns 503 instead of 401 when a dependency (e.g. an auth sidecar) is down.

Diagnostic Workflow

First reproduce the scrape exactly as Prometheus does, from a host on the same network, and read the status line and body:

curl -sS -o /dev/null -w "%{http_code}\n" http://10.0.4.11:8080/metrics
curl -sS -D - http://10.0.4.11:8080/metrics | head -20

A Retry-After or a proxy signature (Server: envoy, nginx) in the response headers tells you whether the app or a proxy produced the 503. Check whether the target is even ready:

# Is the app's own readiness happy while /metrics 503s?
curl -sS -o /dev/null -w "%{http_code}\n" http://10.0.4.11:8080/healthz

On Kubernetes, confirm the Service actually has ready endpoints — an empty endpoint set is a classic source of 503 from the ClusterIP:

kubectl get endpointslices -l kubernetes.io/service-name=api -o wide
kubectl get pods -l app=api -o wide
kubectl describe pod <pod> | sed -n '/Conditions/,/Events/p'

Confirm what Prometheus itself sees, and whether the 503 is flapping with load:

up{job="api"} == 0
sum by (instance) (rate(scrape_samples_scraped{job="api"}[5m]))

If it is intermittent, correlate the 503 windows with target load (request rate, CPU throttling) to confirm load-shedding rather than a hard outage.

Example Root Cause Analysis

A payments API job showed ~30% of its targets DOWN with server returned HTTP status 503 Service Unavailable, only during peak hours.

curl against a failing pod returned:

HTTP/1.1 503 Service Unavailable
Retry-After: 5
Content-Type: text/plain

The Retry-After header pointed at the application, not a proxy. The team’s metrics handler was served by the same synchronous worker pool as API traffic; at peak, the pool was exhausted and the framework’s overload middleware returned 503 for all routes, including /metrics. Prometheus scraping every 15s added measurable load and made the shedding worse.

Fix: they moved metrics onto a dedicated listener (separate goroutine/port) that does not share the request worker pool, so /metrics stays available even when the API sheds load. Scrapes recovered immediately and the metrics needed to diagnose the saturation were no longer the first thing to disappear under saturation.

Prevention Best Practices

• Serve /metrics from a dedicated port or handler that does not share the request worker pool, so metrics survive application overload.
• Ensure the metrics endpoint returns 200 as soon as the process can answer, decoupled from deep readiness checks that legitimately 503.
• Align Kubernetes readinessProbe and rollout maxUnavailable so Services keep ready endpoints during deploys, avoiding LB-level 503s.
• Alert on up == 0 and on elevated 503 scrape errors so a proxy-level 503 is distinguishable from a process outage.
• Give overloaded services a sample_limit and reasonable scrape interval so Prometheus is not amplifying the very saturation it is measuring.
• Point scrape targets at pods/endpoints (via ServiceMonitor/endpoint SD) rather than a ClusterIP that can answer 503 with no backends.

Quick Command Reference

# Reproduce the scrape and read the real status + headers
curl -sS -D - http://TARGET:PORT/metrics | head -20
curl -sS -o /dev/null -w "%{http_code}\n" http://TARGET:PORT/metrics

# Kubernetes: are there ready endpoints behind the Service?
kubectl get endpointslices -l kubernetes.io/service-name=SVC -o wide
kubectl get pods -l app=SVC -o wide

# What Prometheus sees
# up{job="JOB"} == 0
# rate(scrape_samples_scraped{job="JOB"}[5m])

Conclusion

A server returned HTTP status 503 Service Unavailable scrape error means the connection worked but the target deliberately answered 503 — so the fix is never at the network layer. Identify whether the 503 comes from the application (readiness, load-shedding, Retry-After) or from a proxy/Service with no healthy backends, then either let the target become ready, isolate /metrics from request-path saturation, or fix endpoint availability. Alerting separately on 503 scrape errors keeps a proxy problem from looking like a process crash.