🚨 OpenTelemetry Error Guide: 'data refused due to high memory usage' — Tune memory_limiter

Overview

This error is emitted by the OpenTelemetry Collector’s memory_limiter processor when heap usage crosses its configured limit. To protect itself from an out-of-memory kill, the Collector refuses incoming data and pushes backpressure to senders:

2026-07-09T11:07:44.201Z	warn	memorylimiter/memorylimiter.go:222	Memory usage is above soft limit. Refusing data.	{"kind": "processor", "name": "memory_limiter", "cur_mem_mib": 1840}
2026-07-09T11:07:44.203Z	error	internal/queue_sender.go:98	Permanent error: data refused due to high memory usage

data refused due to high memory usage means the memory_limiter is actively shedding load. Refused batches are returned as errors to the receiver, and clients see failed exports until heap usage drops back below the soft limit.

Symptoms

• Clients see RESOURCE_EXHAUSTED / ResourceExhausted or HTTP 503 from the Collector.
• Collector logs repeat Memory usage is above soft limit. Refusing data.
• otelcol_processor_refused_spans / _refused_metric_points climb.
• Collector RSS hovers near its container limit; occasional OOMKilled restarts.
• Throughput drops in bursts as the limiter toggles refusing on and off.

Common Root Causes

• memory_limiter set too low — limit_mib/limit_percentage is below the Collector’s real working set.
• Undersized pod/container memory — the Kubernetes memory limit is too small for the ingest rate.
• No GOMEMLIMIT — the Go runtime grows heap aggressively without a soft memory target.
• Oversized batches / large queues — batch and sending_queue hold too much data in memory at once.
• Backend slowness — a slow exporter causes data to accumulate in memory faster than it drains.
• Cardinality explosion — high-cardinality metrics inflate in-memory state.

Diagnostic Workflow

Configure memory_limiter as the first processor in every pipeline, with check_interval and both hard and soft limits set relative to the container’s memory limit:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500          # hard limit; ~80% of the 2Gi container limit
    spike_limit_mib: 400     # soft-limit headroom for spikes
  batch:
    timeout: 5s
    send_batch_size: 512
    send_batch_max_size: 1024

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]   # memory_limiter FIRST
      exporters: [otlp]

Set a Go soft memory limit so the runtime performs GC before the container OOMs. This complements, not replaces, memory_limiter:

export GOMEMLIMIT=1750MiB     # a bit above limit_mib, below the container limit
export GOGC=80

Watch memory and refusal metrics from the Collector’s telemetry endpoint:

curl -s http://localhost:8888/metrics | grep -E 'otelcol_process_memory_rss|otelcol_processor_refused'

Confirm container limits and any OOM kills in Kubernetes:

kubectl top pod -l app=otel-collector
kubectl describe pod -l app=otel-collector | grep -iA2 'Limits\|OOMKilled\|Last State'

Example Root Cause Analysis

A Collector deployment had a 2Gi container limit but no memory_limiter tuning — it used limit_percentage: 50, capping useful heap at ~1Gi. Under a traffic surge, the batch processor and a large sending_queue pushed the working set past 1Gi, and the limiter began refusing data, returning ResourceExhausted to every exporting service. Meanwhile the container itself was OOMKilled twice because the Go runtime overshot before the limiter could act.

The fix aligned three limits. The container memory limit was raised to 2Gi with a matching request; memory_limiter was set to limit_mib: 1500 with spike_limit_mib: 400; and GOMEMLIMIT=1750MiB was exported so the Go GC reclaimed memory before the hard limit. send_batch_max_size was also capped at 1024 to bound per-batch allocation. RSS stabilized around 1.4Gi, refusals fell to zero outside genuine spikes, and the OOM kills stopped.

Prevention Best Practices

• Always place memory_limiter first in the pipeline so it can shed load before other processors allocate.
• Set limit_mib to roughly 75–80% of the container memory limit, leaving spike_limit_mib headroom.
• Export GOMEMLIMIT slightly above limit_mib so the Go GC and the limiter reinforce each other.
• Bound send_batch_max_size and sending_queue size so memory cannot grow unbounded under load.
• Scale the Collector horizontally (more replicas) rather than one giant instance for high ingest rates.
• Alert on otelcol_processor_refused_* and RSS relative to the limit to catch pressure before refusals.

Quick Command Reference

# Memory and refusal metrics
curl -s http://localhost:8888/metrics | grep -E 'memory_rss|processor_refused'

# Confirm Go soft memory limit is set for the process
tr '\0' '\n' < /proc/$(pgrep -f otelcol)/environ | grep GOMEMLIMIT

# Kubernetes memory usage and OOM history
kubectl top pod -l app=otel-collector
kubectl describe pod -l app=otel-collector | grep -i oomkilled

# Recent limiter warnings
journalctl -u otelcol-contrib | grep -i 'refusing data\|high memory'

Conclusion

data refused due to high memory usage is the memory_limiter doing its job — trading refused batches for a Collector that stays alive instead of being OOMKilled. The lasting fix is to align three numbers: the container memory limit, the memory_limiter limit_mib, and GOMEMLIMIT, with the limiter first in the pipeline and batch/queue sizes bounded. When those agree, the Collector runs near its ceiling without refusing data or crashing, and horizontal scaling handles genuine growth.