🚨 OpenTelemetry Error Guide: 'error reading from server: EOF' — Fix OTLP Dropped Streams

Overview

This error appears when the server or an intermediary closes the OTLP connection while the exporter is still reading the response. The client reaches end-of-file where it expected more data, and gRPC reports it as Unavailable:

rpc error: code = Unavailable desc = error reading from server: EOF

A closely related transport variant reads:

rpc error: code = Unavailable desc = error reading from server: read tcp 10.0.4.12:52788->10.0.7.9:4317: read: connection reset by peer

An unexpected EOF means the far end hung up mid-stream — an idle timeout fired, the connection was recycled, a message exceeded a limit and the server closed the socket, or a load balancer dropped the backend. Because gRPC classifies it as Unavailable, it is retryable: the in-flight batch is retried if a retry queue is configured, otherwise dropped.

Symptoms

• error reading from server: EOF appears intermittently, often after idle periods or during backend rollouts.
• Errors cluster on long-lived connections that have been open for minutes.
• A load balancer, proxy, or service-mesh sidecar sits between the exporter and the Collector.
• Large batches fail with EOF while small ones succeed (server closing on size).
• otelcol_exporter_send_failed_spans blips then recovers as retries land on a healthy connection.

Common Root Causes

• Idle-connection timeout — an LB/NAT/proxy silently closes an idle keepalive connection; the next export reads EOF on a dead socket.
• max_connection_age recycle — the server deliberately closes long-lived HTTP/2 connections and the client sees EOF on the in-flight stream.
• Message too large closing the connection — a batch over the server’s max_recv_msg_size can cause the server to close the socket instead of returning a clean ResourceExhausted.
• Backend/gateway rollout — a Collector replica terminates mid-request and the stream ends in EOF.
• Load balancer draining — the LB removes a backend from rotation and cuts the connection.
• Mismatched keepalive — the client never pings, so a proxy considers the connection idle and closes it between exports.

Diagnostic Workflow

Determine whether EOF follows idle gaps (keepalive/idle-timeout) or large batches (message-size), and what sits in the path:

env | grep OTEL_EXPORTER_OTLP
journalctl -u otelcol-contrib --since '15 min ago' | grep -i 'EOF\|reading from server\|too large\|keepalive'
# Is a mesh sidecar in the pod?
kubectl get pod -l app=my-app -o jsonpath='{.items[0].spec.containers[*].name}'

Keep the connection warm with client keepalive and ensure a retry queue absorbs the occasional recycle:

exporters:
  otlp:
    endpoint: otel-gateway:4317
    tls:
      insecure: false
    keepalive:
      time: 30s               # ping every 30s so idle proxies don't close us
      timeout: 10s
      permit_without_stream: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

On the receiving Collector, permit those pings, recycle connections gracefully, and raise the accepted message size so large batches aren’t closed:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
        keepalive:
          server_parameters:
            max_connection_age: 300s
            max_connection_age_grace: 30s
          enforcement_policy:
            min_time: 20s          # must be <= client keepalive time
            permit_without_stream: true

If the EOF only appears on large exports, cap batch size well under the server limit:

processors:
  batch:
    send_batch_size: 512
    send_batch_max_size: 2048   # keep well under max_recv_msg_size

Then validate and confirm the receiver bound its port:

otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

Example Root Cause Analysis

Application pods exported traces through an L7 load balancer to a gateway Collector. Traffic was bursty: quiet for a minute, then a spike. During the quiet windows the exporter held an idle HTTP/2 connection open but never pinged it, so the load balancer’s 60-second idle timeout closed the backend leg. The next export after each lull read rpc error: code = Unavailable desc = error reading from server: EOF, and because retry_on_failure had been left disabled, that first batch after every idle window was dropped.

The fix had two parts. First, the exporter gained keepalive.time: 30s with permit_without_stream: true so the connection was pinged during idle windows and the LB no longer considered it stale. Second, retry_on_failure and a sending_queue were re-enabled so the rare EOF from a genuine connection recycle became a resumed export. After the change, the post-idle EOFs disappeared and dropped spans went to zero.

Prevention Best Practices

• Enable client keepalive (time ≥ the server’s enforcement_policy.min_time) so idle proxies and LBs don’t close the connection between bursts.
• Always keep retry_on_failure and a bounded sending_queue on so a mid-stream EOF is retried, not dropped.
• Set the server’s max_connection_age with a grace window so long-lived connections are recycled cleanly instead of abruptly.
• Align proxy/LB idle timeouts with the OTLP route and keep them longer than your keepalive interval.
• Cap send_batch_max_size under the receiver’s max_recv_msg_size_mib so oversized batches don’t cause the server to close the socket.
• Alert only on sustained EOF/Unavailable; occasional recycles are normal and should be absorbed by retries.

Quick Command Reference

# Find EOF / mid-stream closes in Collector logs
journalctl -u otelcol-contrib -f | grep -i 'EOF\|reading from server'

# Confirm the gateway is up and speaking gRPC
grpcurl -plaintext otel-gateway:4317 list

# Validate keepalive/message-size config after editing
otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

# Watch failures recover as retries land
curl -s http://localhost:8888/metrics | grep -E 'exporter_send_failed|exporter_sent'

Conclusion

error reading from server: EOF means the connection ended before the response did — the server, a proxy, or a load balancer hung up mid-stream, usually from an idle timeout, a connection recycle, or an oversized batch. Keep the link warm with client keepalive, let the server recycle connections gracefully with max_connection_age, cap batch sizes under the receiver limit, and lean on retry_on_failure so a rare mid-stream close resumes instead of dropping. Reserve alerts for sustained EOFs so normal connection churn stays invisible.

Related

• OpenTelemetry Error Guide: ‘connection reset by peer’ on OTLP export
• OpenTelemetry Error Guide: ‘context deadline exceeded’
• OpenTelemetry Error Guide: ‘connection refused’ to the Collector