🚨 OpenTelemetry Error Guide: 'dial tcp: i/o timeout' on OTLP export — Fix Blocked Connections

Overview

This error appears when an OpenTelemetry exporter tries to open a TCP connection to the Collector and the connect attempt never completes. The SYN packet is sent but no SYN-ACK ever comes back, so after the dial timeout the export fails. It surfaces in SDK and Collector exporter logs:

error	exporterhelper/queue_sender.go:128	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.0.0.9:4317: i/o timeout\"", "dropped_items": 512}

The HTTP/OTLP variant reads:

traces export: Post "https://otel-collector.example.com:4318/v1/traces": dial tcp 10.0.0.9:4318: i/o timeout

dial tcp: i/o timeout means the connect stalled — packets are being silently dropped somewhere in the path. This is different from connection refused (a host answered with RST because nothing listens) and from no such host (DNS never resolved). A timeout almost always points at a firewall, security group, or routing problem swallowing the traffic.

Symptoms

• Every export fails with i/o timeout after a fixed delay, not intermittently.
• The endpoint DNS resolves fine, but the TCP connect never completes.
• curl or nc to the OTLP port hangs until it times out rather than failing fast.
• The problem is environment-specific: works from one subnet/node and times out from another.
• Started after a security-group, NACL, network-policy, or firewall change.
• No RST is ever seen on a packet capture — SYNs go out with no reply.

Common Root Causes

• Firewall or security group drops packets — an AWS security group, GCP firewall, or iptables rule silently discards SYNs to the OTLP port instead of rejecting them, so the client waits out the timeout.
• Wrong or non-routable IP — the endpoint points at an address the host has no route to (a private IP reached from the wrong VPC, or a stale ClusterIP).
• Kubernetes NetworkPolicy — an egress or ingress policy blocks traffic between the app namespace and the Collector.
• Cross-VPC / on-prem routing gap — no peering, transit gateway, or route table entry connects the two networks.
• Collector bound to the wrong interface — the receiver listens on 127.0.0.1 instead of 0.0.0.0, so remote SYNs are dropped by the host.
• Overloaded backlog — the listener’s accept queue is saturated, so new SYNs are dropped and connects time out under load.

Diagnostic Workflow

First distinguish a timeout from a refusal or a DNS failure — they have completely different fixes. A hang is a drop; an instant “refused” is a listening host with a closed port; “no such host” is DNS:

# Resolve the name first — rules out DNS (no such host)
getent hosts otel-collector.example.com

# Timeout = silent drop (firewall/route); instant refusal = port closed
nc -vz -w 5 10.0.0.9 4317
curl -v -m 5 https://otel-collector.example.com:4318/v1/traces

Confirm the receiver is actually bound to a reachable address on the Collector host — a common cause is listening on loopback:

# On the Collector host: is 4317 bound to 0.0.0.0 or just 127.0.0.1?
ss -ltnp | grep -E '4317|4318'

Make sure the receiver endpoint is bound to all interfaces, not loopback:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

From Kubernetes, verify there is no egress NetworkPolicy blocking the path and that the Service resolves to a routable endpoint:

kubectl get networkpolicy -A
kubectl get endpoints otel-collector -n observability
journalctl -u otelcol-contrib --since '15 min ago' | grep -i 'i/o timeout\|dialing'

If the connect is dropped by a cloud firewall, add a rule allowing TCP 4317/4318 from the application’s CIDR to the Collector’s security group, then retest with nc -vz.

Example Root Cause Analysis

After migrating the gateway Collector to a new subnet, an application in 10.0.2.0/24 began logging dial tcp 10.0.0.9:4317: i/o timeout on every export while the old collector kept working. DNS resolved correctly and the collector was healthy, so it was not a refusal or a name-resolution issue. A packet capture on the app node showed SYNs leaving with no SYN-ACK returning — classic silent drop.

The cause was the Collector’s new security group: it only allowed inbound 4317 from the old subnet’s CIDR. The two-part fix was to add an ingress rule permitting TCP 4317/4318 from 10.0.2.0/24, and to confirm the receiver was bound to 0.0.0.0:4317 rather than the node’s old private IP so any in-range client could reach it. After the rule was applied, nc -vz 10.0.0.9 4317 connected instantly and exports resumed with zero drops.

Prevention Best Practices

• Always bind receivers to 0.0.0.0 (or an explicit reachable IP), never 127.0.0.1, for anything beyond a local sidecar.
• Codify security-group / firewall rules for OTLP ports in IaC so a subnet migration can’t silently break connectivity.
• Add a startup connectivity check (nc -vz to the OTLP port) to catch drops before traffic is cut over.
• Use stable DNS names for endpoints, not hard-coded pod or node IPs that change on reschedule.
• Distinguish timeouts from refusals in runbooks — a timeout is a network/firewall problem, a refusal is a listener problem.
• Enable retry_on_failure so a brief routing blip during a migration is retried rather than dropped.

Quick Command Reference

# Rule out DNS first
getent hosts otel-collector.example.com

# Timeout vs refused: hang = drop (firewall/route), instant fail = port closed
nc -vz -w 5 10.0.0.9 4317

# Confirm the receiver is bound to a reachable address
ss -ltnp | grep -E '4317|4318'

# Check for blocking egress/ingress network policies (Kubernetes)
kubectl get networkpolicy -A

# Watch the Collector for dial timeouts
journalctl -u otelcol-contrib -f | grep -i 'i/o timeout\|dialing'

Conclusion

dial tcp: i/o timeout is a silent drop: the exporter’s SYN never gets a reply, so the connect stalls until the timeout fires. The culprit is almost always a firewall, security group, NetworkPolicy, or missing route between the exporter and the Collector — or a receiver bound to loopback. Diagnose by proving it is a timeout and not a refusal or DNS failure, then open the path (security-group rule, route, or 0.0.0.0 bind) and confirm with a fast nc -vz.

Related

• OpenTelemetry Error Guide: ‘connection refused’ to the Collector
• OpenTelemetry Error Guide: ‘no such host’ on OTLP DNS resolution
• OpenTelemetry Error Guide: ‘context deadline exceeded’