🚨 OpenTelemetry Error Guide: 'connection refused' to the Collector — Fix OTLP Endpoints

Overview

This error appears when an OpenTelemetry SDK or an upstream Collector tries to open an OTLP connection to a Collector that is not listening at the target address. It shows up in application logs and in gateway Collectors:

error	exporterhelper/queue_sender.go:128	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.0.4.12:4317: connect: connection refused\""}

The HTTP/protobuf transport variant reads:

traces export: Post "http://otel-collector:4318/v1/traces": dial tcp 10.0.4.12:4318: connect: connection refused

connection refused (ECONNREFUSED) means the TCP handshake reached the host but nothing was listening on that port — the Collector is down, bound to the wrong interface, or the client is pointed at the wrong port.

Symptoms

• No telemetry reaches the backend; the exporting service logs connection refused on every flush.
• The error names port 4317 (gRPC/OTLP) or 4318 (HTTP/OTLP) explicitly.
• The Collector container is restarting, crash-looping, or not yet started.
• kubectl get pods shows the Collector CrashLoopBackOff or 0/1 Ready.
• Curl or grpcurl from the app host to the Collector fails immediately rather than hanging.

Common Root Causes

• Collector not running — crashed on a bad config, OOMKilled, or never scheduled.
• Wrong port — exporting gRPC to 4318 or HTTP to 4317; the two protocols use different ports.
• Receiver not enabled or bound to localhost — the OTLP receiver listens on 127.0.0.1 instead of 0.0.0.0, so remote clients are refused.
• Wrong endpoint host — a stale Service name, IP, or DNS entry after a redeploy.
• NetworkPolicy or firewall — Kubernetes NetworkPolicy or security group blocks the port.
• Protocol mismatch — the SDK uses grpc while only the HTTP receiver is configured (or vice-versa).

Diagnostic Workflow

Confirm what the SDK is targeting. Endpoint and protocol come from environment variables:

echo "$OTEL_EXPORTER_OTLP_ENDPOINT"    # http://otel-collector:4317 (grpc) or :4318 (http)
echo "$OTEL_EXPORTER_OTLP_PROTOCOL"    # grpc | http/protobuf

Ensure the Collector actually enables the OTLP receiver and binds to all interfaces, not loopback:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: normal

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [debug]

Confirm the Collector process is up and the port is listening:

journalctl -u otelcol-contrib --since '10 min ago' | grep -i 'everything is ready\|error\|listening'
ss -lntp | grep -E '4317|4318'

From the client host, test reachability against the exact port the SDK uses:

grpcurl -plaintext otel-collector:4317 list          # gRPC / 4317
curl -v http://otel-collector:4318/v1/traces          # HTTP / 4318

Example Root Cause Analysis

A team migrated their Collector to HTTP/protobuf and set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf but left OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 — the gRPC port. The HTTP exporter dialed 4317, where only the gRPC receiver had been listening; after the Collector’s config was slimmed to HTTP-only, nothing bound 4317 and every export logged connect: connection refused.

The fix was to align the port with the protocol: HTTP/protobuf uses 4318, so the endpoint became http://otel-collector:4318. As a safeguard the Collector’s otlp receiver was configured to enable both grpc (0.0.0.0:4317) and http (0.0.0.0:4318) protocols, so a future protocol switch on any client would not silently break. Telemetry resumed on the next flush.

Prevention Best Practices

• Standardize on one endpoint convention and document which port maps to which protocol: 4317 = gRPC, 4318 = HTTP.
• Always bind Collector receivers to 0.0.0.0 (or the pod IP) in containers, never 127.0.0.1, so remote clients are accepted.
• Add a readiness/liveness probe on the Collector so orchestrators do not route to a Collector that failed to start.
• Enable both OTLP protocols on gateway Collectors to tolerate mixed clients.
• Keep NetworkPolicies explicit: allow ingress to 4317/4318 from the namespaces that export telemetry.
• Alert on the Collector process being down; a refused connection almost always means the receiver is not up.

Quick Command Reference

# Confirm SDK endpoint/protocol
env | grep OTEL_EXPORTER_OTLP

# Verify the Collector is listening on OTLP ports
ss -lntp | grep -E '4317|4318'

# Test reachability by protocol
grpcurl -plaintext otel-collector:4317 list
curl -v http://otel-collector:4318/v1/traces

# Kubernetes: check Collector pod health and logs
kubectl get pods -l app=otel-collector
kubectl logs deploy/otel-collector --tail=50

Conclusion

connection refused is a blunt signal: the TCP port opened but no OTLP receiver answered. Walk it back in order — is the Collector running, is the receiver bound to 0.0.0.0, and does the SDK’s port match its protocol (4317 gRPC, 4318 HTTP)? Enabling both protocols on the Collector and adding readiness probes eliminates the most common variants, so a refused connection becomes a rare, quickly diagnosed event instead of silent telemetry loss.