🚨 Prometheus Error Guide: 'not ready to serve traffic' — Fix Startup 503

Overview

Prometheus returns this body with an HTTP 503 from every endpoint — the UI, /api/v1/query, /federate — while it is still starting up and not yet ready to answer:

Prometheus is not ready to serve traffic. Call /-/ready when ready.

Grafana panels backed by this Prometheus show Bad Gateway or 503, and Kubernetes may restart the pod if the readiness/liveness probe times out. This is expected briefly on every start, but when it persists it usually means WAL replay or TSDB startup is taking far longer than the probe budget, creating a crash loop that never lets Prometheus finish coming up.

Symptoms

• curl http://prometheus:9090/-/ready returns 503 with the “not ready” body.
• Queries and dashboards fail with 503 / Bad Gateway right after a restart or deploy.
• Logs show Replaying WAL, Loading WAL, or TSDB started never being reached before a restart.
• In Kubernetes, the pod cycles Running → Restarting as the readiness probe fails repeatedly.
• prometheus_ready is 0 or the metric is unreachable during the window.

Common Root Causes

• Slow WAL replay — a large write-ahead log takes minutes to replay on startup, exceeding the readiness probe timeout.
• Readiness probe too strict — initialDelaySeconds/failureThreshold too low for the data volume, so Kubernetes kills the pod mid-replay.
• Huge head block / high series churn — startup must reconstruct a large in-memory head.
• Slow storage — the data volume (network disk) has low IOPS, dragging out replay and mmap.
• Corrupt or oversized WAL — replay stalls or repeatedly restarts on a bad segment.
• Cold start after unclean shutdown — an OOMKill or SIGKILL left a large WAL that must be fully replayed.

Diagnostic Workflow

Check readiness and watch the startup logs for the replay phase:

curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9090/-/ready
journalctl -u prometheus --since '10 min ago' | grep -iE 'wal|replay|tsdb|ready'

In Kubernetes, confirm whether the probe is killing the pod before it finishes:

kubectl -n monitoring describe pod prometheus-0 | grep -A5 -iE 'readiness|liveness|last state|restart'

Loosen the readiness probe so replay can complete (raise the failure budget, not the frequency):

readinessProbe:
  httpGet: { path: /-/ready, port: web }
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 60      # ~10 min of grace for large WAL replay
startupProbe:
  httpGet: { path: /-/ready, port: web }
  periodSeconds: 15
  failureThreshold: 80

Once it is up, quantify replay cost so you can size probes correctly next time:

prometheus_tsdb_wal_replay_duration_seconds
prometheus_tsdb_head_series
rate(prometheus_tsdb_head_chunks_created_total[5m])

Example Root Cause Analysis

After an OOMKill, a Prometheus pod entered a crash loop, always logging Replaying WAL and then restarting. The readiness probe had failureThreshold: 6 at periodSeconds: 10 — only 60 seconds of grace — but the WAL for ~4M active series took roughly 7 minutes to replay on the network-backed volume.

Kubernetes killed the pod at 60s, every time, so replay never finished. Raising the readiness failureThreshold and adding a startupProbe with a generous budget let replay complete; prometheus_tsdb_wal_replay_duration_seconds recorded ~410s. The durable fix was also to reduce head series (cardinality controls) and move to a higher-IOPS disk so future replays finish in under two minutes.

Prevention Best Practices

• Give Prometheus a startupProbe with a budget larger than your worst-case WAL replay time, so liveness/readiness never kill a healthy-but-replaying pod.
• Track prometheus_tsdb_wal_replay_duration_seconds and alert when it trends up — it predicts future startup pain.
• Control head cardinality; fewer active series means faster replay and startup.
• Use storage with adequate IOPS for the data volume; replay is I/O bound.
• Prevent OOMKills (memory limits + cardinality control) so restarts aren’t forced with a large unclean WAL.

Quick Command Reference

# Readiness + startup logs
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9090/-/ready
journalctl -u prometheus --since '10 min ago' | grep -iE 'wal|replay|ready'

# Kubernetes probe / restart inspection
kubectl -n monitoring describe pod prometheus-0 | grep -A5 -iE 'readiness|restart'

prometheus_tsdb_wal_replay_duration_seconds
prometheus_tsdb_head_series

Conclusion

“not ready to serve traffic” is normal for a few seconds at startup and a problem only when it persists — almost always because WAL replay outlasts a too-strict readiness probe, creating a crash loop. Give replay enough grace with a startupProbe, measure replay duration, and attack the root cause (cardinality, disk IOPS, OOMKills) so startups stay short.