🚨 OpenTelemetry Error Guide: '503 Service Unavailable' on OTLP export — Fix Backend Outages

Overview

This error appears when the OTLP backend or the gateway in front of it is temporarily down, overloaded, or has no healthy upstream. The server returns HTTP 503 and the Collector’s exporter logs the failure:

error exporting items, request to https://otel.example.com/v1/traces responded with HTTP Status Code 503, Message=Service Unavailable

A load balancer with no ready backends produces the same status from its own layer:

2026/07/12 14:22:10 [error] 41#41: *9021 no live upstreams while connecting to upstream, client: 10.0.0.11, request: "POST /v1/traces HTTP/1.1", upstream: "http://otel_backend", host: "otel.example.com"

HTTP 503 is transient by definition: the request was valid but the service could not handle it right now, so it should be retried with backoff — and buffered durably so a longer outage does not become data loss.

Symptoms

• Export failures cluster during backend deploys, restarts, or capacity events, then clear on their own.
• Collector logs repeat HTTP Status Code 503, Message=Service Unavailable on the otlphttp or otlp exporter.
• A load balancer or ingress in front of the backend reports no live upstreams or failing health checks.
• Telemetry gaps line up exactly with backend maintenance windows.
• otelcol_exporter_send_failed_spans spikes during the outage and returns to zero afterward.

Common Root Causes

• Backend rolling deploy — all replicas are briefly unready during an upgrade, so the gateway has no healthy target.
• Overloaded backend — the ingest tier sheds load with 503 when saturated rather than accepting more data.
• Gateway with no upstreams — the load balancer’s health checks fail and it returns 503 from its own layer.
• No retry configured — a transient 503 immediately drops the batch instead of waiting for recovery.
• Memory-only queue — an in-memory sending_queue cannot survive a Collector restart during a long outage.
• Undersized queue — the buffer overflows before the backend comes back, turning a blip into dropped spans.

Diagnostic Workflow

Confirm the 503 is transient and check whether the backend or the gateway is the source:

journalctl -u otelcol-contrib --since '15 min ago' | grep -i '503\|service unavailable\|no live upstreams'

# Probe the backend health endpoint directly
curl -s -o /dev/null -w '%{http_code}\n' https://otel.example.com/v1/traces
curl -s https://otel.example.com/healthz

Configure retry with backoff and a persistent, file-backed sending queue so batches survive both the outage and a Collector restart:

extensions:
  file_storage:
    directory: /var/lib/otelcol/queue   # persist queued batches to disk

exporters:
  otlphttp:
    endpoint: https://otel.example.com
    compression: gzip
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 60s
      max_elapsed_time: 900s      # keep retrying across a 15-min outage
    sending_queue:
      enabled: true
      storage: file_storage       # durable queue survives restarts
      num_consumers: 4
      queue_size: 20000

service:
  extensions: [file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Add the Collector’s own health check so orchestration and monitoring can distinguish a Collector fault from a downstream backend outage:

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check, file_storage]

# Collector liveness — 200 means the Collector is up even if the backend is 503-ing
curl -s http://localhost:13133/status

Example Root Cause Analysis

A vendor backend ran a rolling upgrade every night at 02:00. For ~90 seconds its gateway had no ready upstreams and returned 503 to every OTLP request. The Collector used a default in-memory sending_queue with retry_on_failure disabled, so it logged HTTP Status Code 503, Message=Service Unavailable and dropped every batch for the duration — a nightly hole in the traces.

The fix had two parts. First, retry_on_failure was enabled with max_elapsed_time: 900s, so batches were held and re-sent as soon as the backend returned rather than dropped on the first 503. Second, the sending_queue was backed by file_storage with queue_size: 20000, so the 90-second backlog spilled to disk and even a Collector restart mid-window would not lose it. The nightly gap disappeared; queued spans flushed within seconds of the backend recovering.

Prevention Best Practices

• Enable retry_on_failure with a max_elapsed_time long enough to ride out your backend’s worst realistic outage.
• Back the sending_queue with file_storage so a restart during an outage does not discard buffered telemetry.
• Size queue_size to hold the volume produced across a full maintenance window, with margin.
• Run the health_check extension so monitoring separates Collector faults from downstream 503s.
• Coordinate backend maintenance windows with alert suppression so transient 503 retries do not page on-call.
• Alert only on sustained send_failed_spans, not on brief spikes that retries will absorb.

Quick Command Reference

# Watch for transient 503s and upstream failures
journalctl -u otelcol-contrib -f | grep -i '503\|unavailable\|no live upstreams'

# Check Collector liveness independent of the backend
curl -s http://localhost:13133/status

# Confirm the persistent queue directory is being used
ls -la /var/lib/otelcol/queue

# Validate config after adding retry/persistent queue
otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

Conclusion

A 503 Service Unavailable on OTLP export means the backend is temporarily down or overloaded, not that your telemetry is malformed. Because it is transient, the right response is patience with durability: retry_on_failure with backoff to wait out the outage, plus a file_storage-backed sending_queue so batches survive both the wait and a Collector restart. Add a health_check so you can tell a Collector problem from a downstream one, and reserve alerts for outages that outlast your retry window.

Related

• OpenTelemetry Error Guide: ‘no more retries left; dropping data’
• OpenTelemetry Error Guide: ‘sending queue is full’
• OpenTelemetry Error Guide: ‘connection refused’ to the Collector