🚨 OpenTelemetry Error Guide: 'rpc error: code = Unavailable' — Fix OTLP gRPC Connectivity

Overview

This error appears when the OTLP gRPC exporter cannot establish or keep a working transport to the Collector or backend. gRPC reports Unavailable for transient transport problems — the endpoint is down, a load balancer is draining, or the network briefly dropped. It shows up in SDK and Collector exporter logs:

rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.0.0.9:4317: connect: connection refused"

A drain/rollout variant reads:

rpc error: code = Unavailable desc = the server is currently unavailable

Unavailable is defined by gRPC as a retryable condition: the call could not reach a healthy server this time, but the same request may well succeed on retry. Unlike a Permanent error, the exporter’s retry_on_failure will keep attempting delivery.

Symptoms

• Bursts of code = Unavailable in exporter logs that recover on their own after a short backoff.
• Errors align with Collector/gateway deployments, pod restarts, or load-balancer draining events.
• dial tcp ...: connect: connection refused in the desc when the target has no listener at that moment.
• otelcol_exporter_send_failed_spans spikes briefly, then otelcol_exporter_sent_spans resumes.
• Steady-state export works, but any endpoint churn produces a short gap.

Common Root Causes

• Collector is down or restarting — the target process isn’t listening yet (or anymore), so the dial is refused.
• Load balancer draining — the LB pulled a backend out of rotation mid-connection and new streams have nowhere to land.
• Network blip / partition — a transient route or firewall state drop breaks the TCP dial for a few seconds.
• DNS points to a stale IP — a scaled-down replica’s address is still cached, so the exporter dials a dead host.
• Readiness before listener — the Collector’s service is announced before the OTLP receiver has bound its port.
• No retry headroom — retry_on_failure is disabled, so a momentary Unavailable becomes a permanent drop.

Diagnostic Workflow

Confirm what the exporter targets and whether anything is listening there right now:

echo "$OTEL_EXPORTER_OTLP_ENDPOINT"        # e.g. http://otel-collector:4317
grpcurl -plaintext 10.0.0.9:4317 list      # should list OTLP services
nc -vz 10.0.0.9 4317                       # is the port open at all?

Ensure the exporter has retry and a bounded queue so transient Unavailable is absorbed rather than dropped:

exporters:
  otlp:
    endpoint: otel-collector:4317
    tls:
      insecure: false
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Check the receiving Collector actually bound its gRPC port and did not recently crash-loop:

journalctl -u otelcol-contrib --since '15 min ago' | grep -i 'unavailable\|dialing\|Everything is ready\|listening'
kubectl get pods -l app=otel-collector    # look for restarts / NotReady

If a load balancer is in the path, verify its health checks target the OTLP port and that draining connections are handled gracefully by client retries.

Example Root Cause Analysis

A fleet of application pods exported traces to a Collector Service fronting three gateway replicas. During a routine rolling update, one replica was terminated while exporters still held streams to it; new dials briefly hit a pod that had stopped listening and returned rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.0.0.9:4317: connect: connection refused". Because retry_on_failure had been disabled in a prior “simplify the config” change, every in-flight batch during the 8-second rollout window was dropped instead of retried.

The fix had two parts. First, retry_on_failure and a sending_queue of 5000 were restored on the exporter so a dial failure backed off and retried against a healthy replica. Second, the gateway Deployment gained a preStop sleep and maxUnavailable: 0 so terminating replicas drained gracefully before the port closed. After the change, subsequent rollouts produced a handful of retried Unavailable log lines and zero dropped spans.

Prevention Best Practices

• Always keep retry_on_failure enabled with a bounded sending_queue so transient Unavailable never drops data.
• Roll Collectors with maxUnavailable: 0 and a preStop drain delay so terminating replicas stop accepting new streams cleanly.
• Point exporters at a stable Service/DNS name, not a pod IP, so scale events don’t leave you dialing dead hosts.
• Configure LB health checks against the actual OTLP port so unhealthy backends leave rotation before clients hit them.
• Deploy a local/agent Collector so applications export over a short, stable hop and the gateway churn is hidden behind retries.
• Alert on sustained Unavailable (minutes), not single blips, so rollouts don’t page you.

Quick Command Reference

# Is anything listening on the OTLP gRPC port right now?
grpcurl -plaintext otel-collector:4317 list
nc -vz otel-collector 4317

# Watch for transient Unavailable / dialing errors
journalctl -u otelcol-contrib -f | grep -i 'Unavailable\|dialing'

# Check the receiving Collector for restarts
kubectl get pods -l app=otel-collector

# Confirm retries are absorbing failures (sent resumes after failed spikes)
curl -s http://localhost:8888/metrics | grep -E 'exporter_send_failed|exporter_sent'

Conclusion

rpc error: code = Unavailable is gRPC’s way of saying “not right now” — the Collector or backend was unreachable for this attempt, usually because of a restart, a draining load balancer, or a brief network hiccup. Because it is retryable, the durable fix is rarely at the network layer alone: keep retry_on_failure and a sending_queue enabled so momentary unavailability is absorbed, drain Collectors gracefully during rollouts, and target stable service names. Reserve alerts for sustained unavailability so normal churn stays invisible.

Related

• OpenTelemetry Error Guide: ‘connection refused’ to the Collector
• OpenTelemetry Error Guide: ‘context deadline exceeded’
• OpenTelemetry Error Guide: ‘no such host’