🚨 OpenTelemetry Error Guide: 'write: broken pipe' on OTLP export — Fix Dropped Connections

Overview

This error appears when an OpenTelemetry exporter tries to write bytes onto a TCP connection that the peer has already closed. The socket looked healthy at the start of the request, but before the write completed the far end sent a FIN or RST, so the kernel fails the write() syscall. It surfaces in SDK and Collector exporter logs:

error	exporterhelper/queue_sender.go:128	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = write tcp 10.0.0.5:52344->10.0.0.9:4317: write: broken pipe", "dropped_items": 512}

The HTTP/OTLP variant reads similarly:

traces export: Post "https://otel-gateway.example.com:4318/v1/traces": write tcp 10.0.0.5:52344->10.0.0.9:4318: write: broken pipe

broken pipe (EPIPE) means the connection existed and was torn down by the peer mid-write — the Collector restarted, a load balancer culled an idle connection, or a proxy reset the stream. The in-flight batch is dropped and retried only if a retry queue is configured.

Symptoms

• Intermittent write: broken pipe errors that come and go rather than failing every export.
• Failures cluster right after a downstream Collector rollout, deploy, or scale-in event.
• Errors appear after quiet periods, when a keepalive connection has gone idle and been reaped.
• A load balancer, NAT gateway, or service-mesh sidecar sits between the exporter and the Collector.
• gRPC clients also log transport is closing or Unavailable alongside the broken-pipe messages.
• Small exports occasionally fail while the pipeline is otherwise healthy in steady state.

Common Root Causes

• Collector restarted mid-write — a rolling update or crash on the downstream Collector closes open sockets while a batch is being sent.
• Load balancer idle timeout — an LB, NAT, or proxy silently drops an idle keepalive connection; the next write lands on a dead socket and returns EPIPE.
• Missing or too-slow keepalive — without periodic pings the client never learns the connection is stale until it tries to write.
• Proxy stream limits — an L7 proxy (Envoy, nginx, ALB) enforces max request duration or max streams and closes the connection under the exporter.
• No retry queue — a single transient reset becomes permanent data loss because nothing re-sends the dropped batch.
• Connection reuse across a restart — a pooled HTTP/2 connection is reused after the server already sent GOAWAY.

Diagnostic Workflow

Confirm what the exporter is actually connecting to and whether a proxy or sidecar sits in the path:

# What endpoint is the SDK/Collector exporting to?
env | grep OTEL_EXPORTER_OTLP
# Is a mesh sidecar injected into the pod?
kubectl get pod -l app=my-app -o jsonpath='{.items[0].spec.containers[*].name}'

Check whether the downstream Collector restarted around the time of the errors:

kubectl get pods -l app=otel-gateway            # look for recent restarts / churn
journalctl -u otelcol-contrib --since '15 min ago' | grep -i 'broken pipe\|dropping data\|shutting down'

Enable client keepalive so idle connections stay warm and dead ones are detected before the next write, and add a retry queue so a single reset is re-sent rather than dropped:

exporters:
  otlp:
    endpoint: otel-gateway.example.com:4317
    tls:
      insecure: false
    keepalive:
      time: 30s               # ping every 30s to keep the path alive
      timeout: 10s
      permit_without_stream: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

On the receiving Collector, keep connection ages bounded and permit the client’s pings so the server recycles connections gracefully instead of leaving half-dead sockets:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        keepalive:
          server_parameters:
            max_connection_age: 30s
            max_connection_age_grace: 5s
          enforcement_policy:
            min_time: 20s          # must be <= client keepalive time
            permit_without_stream: true

Validate the config before reloading and watch the exporter’s self-telemetry for send failures:

otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml
curl -s http://localhost:8888/metrics | grep -E 'otelcol_exporter_send_failed|otelcol_exporter_sent'

Example Root Cause Analysis

A fleet of application pods exported traces through an internal AWS Network Load Balancer to a gateway Collector. During off-peak hours traffic was sparse, and the NLB’s default 350-second idle timeout reaped the exporters’ pooled gRPC connections. Because the SDK had no keepalive configured, it never noticed; the next export reused the dead connection and the kernel returned write tcp 10.0.0.5:52344->10.0.0.9:4317: write: broken pipe. Roughly 8% of spans vanished each morning as traffic resumed.

The fix had two parts. First, keepalive was enabled on the exporter (time: 30s, permit_without_stream: true) so the connection was pinged well inside the NLB idle window and stale sockets were detected before any write. Second, retry_on_failure with a bounded sending_queue was added so the rare reset during a gateway rollout was re-sent instead of dropped. After the change the morning span loss fell to zero, and the only remaining resets — during deliberate gateway deploys — were transparently retried.

Prevention Best Practices

• Always configure client keepalive with a time shorter than every idle timeout in the network path (LB, NAT, proxy, mesh).
• Enable retry_on_failure and a bounded sending_queue so a mid-write reset becomes a resumed export, not lost data.
• Bound the server’s max_connection_age so connections are recycled predictably rather than dying unexpectedly.
• Raise or disable the OTLP route’s idle timeout on any L7 load balancer or mesh in the path.
• Expect resets during rollouts; alert only on sustained broken-pipe rates, not the occasional deploy blip.
• Prefer a local gateway Collector so applications write over a short-lived local hop instead of a long-haul connection through multiple proxies.

Quick Command Reference

# Inspect the exporter endpoint the SDK/Collector uses
env | grep OTEL_EXPORTER_OTLP

# Watch for broken-pipe / drop events live
journalctl -u otelcol-contrib -f | grep -i 'broken pipe\|dropping data'

# Check whether the downstream Collector restarted recently
kubectl get pods -l app=otel-gateway

# Scrape the Collector's own metrics for send failures
curl -s http://localhost:8888/metrics | grep otelcol_exporter_send_failed

# Validate config after adding keepalive/retry settings
otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

Conclusion

write: broken pipe means the OTLP connection existed and was torn down by the peer while the exporter was still writing — almost always a Collector restart or an idle-connection timeout in a load balancer or proxy. The durable fix pairs client keepalive (so stale connections are detected and refreshed before a write) with retry_on_failure and a sending_queue (so the rare mid-flight reset is re-sent rather than dropped). Align keepalive with every idle timeout in the path and the broken pipes disappear.

Related

• OpenTelemetry Error Guide: ‘connection reset by peer’ on OTLP export
• OpenTelemetry Error Guide: ‘connection refused’ to the Collector
• OpenTelemetry Error Guide: ‘context deadline exceeded’