🚨 OpenTelemetry Error Guide: 'ResourceExhausted: received message larger than max' — Fix gRPC OTLP Size Limits

Overview

The OTLP gRPC exporter and receiver enforce a maximum message size. When a batch of spans, metrics, or logs serializes larger than that limit, gRPC rejects the whole message and the SDK or upstream Collector logs a ResourceExhausted error:

Exporting failed. The error is not retryable. Dropping data.
error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5242880 vs. 4194304)

The same limit shows up as a permanent send failure in a Collector-to-Collector or Collector-to-backend hop:

Permanent error: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (7340032 vs. 4194304)

Because the error is classified as non-retryable, the exporter drops the batch rather than queuing it, so the data is lost permanently.

Symptoms

• Spans, metrics, or logs disappear intermittently, correlated with high-volume services or large batches.
• Collector or SDK logs contain code = ResourceExhausted with a (X vs. 4194304) size comparison (4 MiB is the gRPC default).
• Small/low-volume services export fine; only busy services or wide spans fail.
• otelcol_exporter_send_failed_spans climbs while otelcol_exporter_sent_spans stalls.
• Errors appear right after enabling large attributes, big log bodies, or a higher batch size.

Common Root Causes

• Batch larger than the receiver limit — the batch processor’s send_batch_max_size produces messages over the receiver’s max_recv_msg_size_mib.
• Default 4 MiB gRPC ceiling — neither side raised the default, but payloads grew.
• Oversized individual telemetry — huge span attributes, stack traces in events, or large log bodies push a single item near the limit.
• Fan-in at a gateway — many agents batching into one gateway multiply message size.
• Metrics with high cardinality — a single scrape/export carrying thousands of series exceeds the limit.

Diagnostic Workflow

First confirm the limit and direction from the log line: received means the receiver rejected it, send/trying to send means the exporter refused to transmit. The numbers are (actual vs. allowed) in bytes.

Check the sending side’s batch configuration in the Collector:

processors:
  batch:
    send_batch_size: 8192
    send_batch_max_size: 8192   # hard cap on items per exported batch
    timeout: 5s

Raise the receiver’s message size limit on the destination Collector so it accepts larger payloads:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16   # default is 4

If the exporter is the one refusing to send, the OTLP exporter does not expose a send-size knob in all versions, so the fix is to shrink the batch. For SDK exporters, cap batch size with environment variables:

export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=256
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

Inspect the Collector’s own telemetry to see failures and confirm the fix:

otelcol --config /etc/otelcol/config.yaml   # startup validates config
journalctl -u otelcol -f | grep -i 'ResourceExhausted\|larger than max'
curl -s http://localhost:8888/metrics | grep -E 'send_failed|refused'

Example Root Cause Analysis

A payments service started dropping traces after a release that added full request/response bodies as span attributes. The gateway Collector logged received message larger than max (6291456 vs. 4194304). The agent’s batch processor used send_batch_max_size: 8192, and with the new fat attributes each batch serialized to ~6 MiB against the gateway’s default 4 MiB receiver limit.

Two changes fixed it: the gateway’s OTLP receiver was raised to max_recv_msg_size_mib: 16, and the agent’s send_batch_max_size was lowered to 2048 so batches stayed well under the ceiling. The team also trimmed the oversized attributes with a transform processor, since request bodies did not belong on spans. After the change, otelcol_exporter_send_failed_spans dropped to zero and traces reappeared.

Prevention Best Practices

• Set send_batch_max_size on every batch processor so no batch can grow unbounded; keep it comfortably below the receiver limit.
• Raise max_recv_msg_size_mib on receiving Collectors when you legitimately need larger batches, and keep sender and receiver limits consistent across every hop.
• Keep individual telemetry small: strip large request/response bodies, cap attribute value length, and avoid dumping full payloads onto spans.
• Alert on otelcol_exporter_send_failed_spans and ResourceExhausted in logs so silent drops surface immediately.
• Load-test the pipeline at peak volume before rollout so size limits are hit in staging, not production.

Quick Command Reference

# Validate config on startup
otelcol validate --config /etc/otelcol/config.yaml

# Watch for the error live
journalctl -u otelcol -f | grep -i 'larger than max'

# Check exporter failure/refused counters
curl -s http://localhost:8888/metrics | grep -E 'send_failed_spans|refused_spans'

# Cap SDK batch size to shrink messages
export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=256

Conclusion

ResourceExhausted: received message larger than max is a size-limit mismatch: a serialized OTLP batch exceeded the gRPC message ceiling and, because the error is non-retryable, the data was dropped. Fix it from both ends — cap send_batch_max_size (or the SDK batch size) on the sender and raise max_recv_msg_size_mib on the receiver — and keep individual telemetry lean. Aligning batch size and receiver limits across every hop keeps busy services from silently losing data.