🚨 OpenTelemetry Error Guide: 'connection reset by peer' on OTLP export — Fix Dropped Streams

Overview

This error appears when an established OTLP connection is abruptly closed by the other end mid-request. Unlike connection refused (nothing was listening) it means a connection did open and was then torn down. It shows up in SDK and Collector exporter logs:

error	exporterhelper/queue_sender.go:128	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = error reading from server: read tcp 10.0.4.12:52344->10.0.7.9:4317: read: connection reset by peer"}

The HTTP/OTLP variant reads:

traces export: Post "https://otel-gateway:4318/v1/traces": read tcp 10.0.4.12:52344->10.0.7.9:4318: read: connection reset by peer

connection reset by peer (ECONNRESET) means the remote side sent a TCP RST — the Collector, a load balancer, or a proxy between them killed the socket. The batch in flight is dropped and retried if a retry queue is configured.

Symptoms

• Intermittent connection reset by peer / Unavailable errors that come and go rather than failing every export.
• Errors cluster after periods of low traffic (idle connections) or during backend/gateway rollouts.
• A load balancer or service mesh sidecar sits between the exporter and the Collector.
• gRPC clients log GOAWAY or transport is closing alongside the resets.
• Exports of large batches fail with a reset while small ones succeed.

Common Root Causes

• Idle-connection timeout — an LB, NAT, or proxy silently drops idle keepalive connections; the next export hits a dead socket and gets a reset.
• Gateway rollout — the downstream Collector replica is terminated and sends RST/GOAWAY to open streams.
• LB / proxy request limits — an L7 proxy (Envoy, nginx, ALB) enforces max request duration, max concurrent streams, or max header/body size and resets offenders.
• HTTP/2 max-concurrent-streams exhaustion on the server, causing GOAWAY and reset streams.
• Message too large — a batch exceeding the server’s max_recv_msg_size can surface as a reset rather than a clean ResourceExhausted.
• Mismatched keepalive — the client’s keepalive is more aggressive than the server’s keepalive.enforcement_policy, so the server resets the connection.

How to diagnose

Confirm whether a proxy/LB sits in the path and what its idle timeout is — resets after idle periods almost always point here:

# What is the exporter actually connecting to?
env | grep OTEL_EXPORTER_OTLP
# Is a mesh sidecar in the pod?
kubectl get pod -l app=my-app -o jsonpath='{.items[0].spec.containers[*].name}'

Check the Collector’s keepalive and message-size settings and whether it recently restarted:

kubectl logs deploy/otel-gateway --previous | grep -i 'shutting down\|keepalive\|too large'
kubectl get pods -l app=otel-gateway   # look for recent restarts / churn

Watch export failures on the client’s self-telemetry to see if resets correlate with batch size or idle gaps:

curl -s http://localhost:8888/metrics | grep -E 'otelcol_exporter_send_failed|otelcol_exporter_sent'

Fixes

Tune client keepalive so idle connections are kept warm and dead ones are detected quickly. On the OTLP gRPC exporter:

exporters:
  otlp:
    endpoint: otel-gateway:4317
    tls:
      insecure: false
    keepalive:
      time: 30s               # ping every 30s to keep the path alive
      timeout: 10s
      permit_without_stream: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

On the receiving Collector, allow the client’s keepalive pings and raise the accepted message size so large batches aren’t reset:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
        keepalive:
          server_parameters:
            max_connection_idle: 11s
            max_connection_age: 30s
            max_connection_age_grace: 5s
          enforcement_policy:
            min_time: 20s          # must be <= client keepalive time
            permit_without_stream: true

If an L7 load balancer or mesh is in the path, raise or disable its idle timeout for the OTLP route and confirm its max request/stream limits accommodate your batch sizes. For AWS NLB/ALB, increase the idle timeout; for Envoy, set a generous idle_timeout and adequate max_concurrent_streams on the cluster.

Cap batch size so a single export never exceeds the server limit:

processors:
  batch:
    send_batch_size: 512
    send_batch_max_size: 2048   # keep well under max_recv_msg_size

What to watch out for

• The client keepalive time must be greater than or equal to the server’s enforcement_policy.min_time, or the server treats pings as abuse and resets you — a common self-inflicted cause.
• Resets during rollouts are expected; a retry queue with persistence turns them into resumed exports rather than lost data. Alert only on sustained resets.
• A reset that only happens on large batches is a message-size or proxy-body-limit problem, not a keepalive problem — fix the size, not the timeouts.
• Behind a service mesh, mTLS renegotiation and sidecar restarts also produce resets; check the sidecar, not just the Collector.
• Do not confuse connection reset by peer (connection existed, then died) with connection refused (nothing listening) or no such host (DNS) — they have different fixes.

Related

• OpenTelemetry Error Guide: ‘connection refused’ to the Collector
• OpenTelemetry Error Guide: ‘no such host’
• OpenTelemetry Error Guide: ‘context deadline exceeded’