🚨 OpenTelemetry Error Guide: 'stream terminated by RST_STREAM' — Fix OTLP gRPC/HTTP2 Resets

Overview

This error appears when the HTTP/2 stream carrying an OTLP request is reset by the server, a proxy, or a load balancer before the export completes. gRPC surfaces the underlying RST_STREAM frame and wraps it as Unavailable:

rpc error: code = Unavailable desc = stream terminated by RST_STREAM with error code: ENHANCE_YOUR_CALM

A plain reset without the flood-protection code looks like:

rpc error: code = Unavailable desc = stream terminated by RST_STREAM with error code: NO_ERROR

RST_STREAM is an HTTP/2 frame that cancels a single stream. ENHANCE_YOUR_CALM is the server’s explicit “you are sending too fast / pinging too aggressively” signal — it fires when a proxy’s flood protection, idle limit, or max_connection_age trips. The affected batch is dropped and retried if a retry queue is configured.

Symptoms

• stream terminated by RST_STREAM appears intermittently, often clustered rather than on every export.
• The ENHANCE_YOUR_CALM code shows up when a client keepalive is more aggressive than the server permits.
• Errors line up with a proxy/LB (Envoy, nginx, ALB) or service-mesh sidecar in the OTLP path.
• Long-lived connections fail at a regular interval matching a max_connection_age.
• Resets increase under high export concurrency as max_concurrent_streams is exhausted.

Common Root Causes

• Keepalive too aggressive — the client pings more often than the server’s enforcement_policy.min_time allows, so the server answers with ENHANCE_YOUR_CALM and resets the stream.
• max_connection_age reached — the server deliberately recycles long-lived HTTP/2 connections and RSTs in-flight streams during the cutover.
• Proxy/LB flood protection — Envoy or an L7 LB caps request rate or stream count and resets excess streams.
• Idle-stream timeout — the proxy resets streams that sit idle longer than its configured limit.
• max_concurrent_streams exhaustion — too many parallel exports on one connection exceed the server/proxy stream cap, forcing resets.
• HTTP/2 downgrade or mismatch — an intermediary that mishandles HTTP/2 prior-knowledge sends spurious RST_STREAM frames.

Diagnostic Workflow

Identify whether the reset carries ENHANCE_YOUR_CALM (a rate/keepalive problem) or NO_ERROR (a connection-age/idle recycle), and what sits in the path:

journalctl -u otelcol-contrib --since '15 min ago' | grep -i 'RST_STREAM\|ENHANCE_YOUR_CALM\|keepalive'
# Is a mesh sidecar in the pod?
kubectl get pod -l app=my-app -o jsonpath='{.items[0].spec.containers[*].name}'

Relax the client’s keepalive so it never pings faster than the server allows, and cap parallelism so streams don’t exhaust the connection:

exporters:
  otlp:
    endpoint: otel-gateway:4317
    tls:
      insecure: false
    keepalive:
      time: 30s                 # must be >= server enforcement_policy.min_time
      timeout: 10s
      permit_without_stream: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

On the receiving Collector, align the server keepalive so it tolerates the client’s ping rate and recycles connections gracefully:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
        keepalive:
          server_parameters:
            max_connection_age: 300s
            max_connection_age_grace: 30s
          enforcement_policy:
            min_time: 20s          # must be <= client keepalive time
            permit_without_stream: true

If an L7 proxy or mesh is in the path, raise its idle-stream timeout and max_concurrent_streams for the OTLP route so bursts of parallel exports aren’t reset. Then validate:

otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

Example Root Cause Analysis

An agent Collector was tuned for “fresh connections” with keepalive.time: 10s to detect dead gateways quickly. The gateway Collector, however, enforced min_time: 20s. Every time the agent’s connection sat without an active stream, it pinged at 10-second intervals — twice as fast as allowed — and the gateway responded with rpc error: code = Unavailable desc = stream terminated by RST_STREAM with error code: ENHANCE_YOUR_CALM. Because the resets came in bursts, roughly 6% of batches were retried and export latency climbed.

The fix had two parts. First, the agent’s keepalive.time was raised to 30s so it never pinged faster than the gateway’s min_time, and permit_without_stream: true was set on both ends so idle pings were allowed. Second, the gateway’s max_connection_age was extended to 300s with a 30s grace so long-lived connections were recycled cleanly rather than tearing down active streams. After the change, ENHANCE_YOUR_CALM disappeared and retried batches dropped to zero.

Prevention Best Practices

• Keep client keepalive time greater than or equal to the server’s enforcement_policy.min_time — the single most common cause of ENHANCE_YOUR_CALM.
• Set permit_without_stream: true on both client and server so idle keepalive pings are permitted instead of punished.
• Give the server a generous max_connection_age plus max_connection_age_grace so connection recycling drains streams cleanly.
• Raise proxy/LB max_concurrent_streams and idle-stream timeouts on the OTLP route to match your export concurrency.
• Keep retry_on_failure and a persistent sending_queue enabled so any reset becomes a resumed export, not lost data.
• Alert only on sustained resets; occasional NO_ERROR RSTs during connection recycling are expected.

Quick Command Reference

# Distinguish ENHANCE_YOUR_CALM (rate) from NO_ERROR (recycle) resets
journalctl -u otelcol-contrib -f | grep -i 'RST_STREAM'

# Validate keepalive config after editing
otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

# Confirm the gateway is reachable and speaking gRPC
grpcurl -plaintext otel-gateway:4317 list

# Watch export failures vs. successful sends
curl -s http://localhost:8888/metrics | grep -E 'exporter_send_failed|exporter_sent'

Conclusion

stream terminated by RST_STREAM means an HTTP/2 stream was cancelled mid-flight — and ENHANCE_YOUR_CALM specifically means the far end thinks you are too aggressive. The fix is almost always keepalive alignment: make the client’s ping interval at least the server’s min_time, permit pings without an active stream, and let the server recycle connections with a generous max_connection_age and grace. Back it with a retry queue so the rare legitimate reset resumes instead of dropping, and reserve alerts for sustained storms rather than routine recycling.

Related

• OpenTelemetry Error Guide: ‘connection reset by peer’ on OTLP export
• OpenTelemetry Error Guide: ‘sending queue is full’
• OpenTelemetry Error Guide: ‘context deadline exceeded’