🚨 OpenTelemetry Error Guide: '429 Too Many Requests' on OTLP export — Fix Backend Rate Limiting

Overview

This error appears when the OTLP backend rate-limits the Collector: you are sending telemetry faster than your quota or the ingest tier allows, so the server answers HTTP 429 and the exporter logs the failure:

error exporting items, request to https://otel.example.com/v1/traces responded with HTTP Status Code 429, Message=Too Many Requests

Well-behaved backends pair the 429 with a Retry-After header telling the client how long to wait before retrying:

2026-07-12T14:22:10.512Z	info	otlphttp	Retry-After header received	{"kind": "exporter", "data_type": "traces", "name": "otlphttp", "interval": "30s"}

HTTP 429 means the request was well-formed but throttled — the backend is protecting itself, and the batch should be retried with backoff rather than resent immediately.

Symptoms

• Telemetry arrives in bursts then stalls, tracking your billing plan’s rate ceiling.
• Collector logs repeat HTTP Status Code 429, Message=Too Many Requests on the otlphttp or otlp exporter.
• Errors intensify at deploy time or during traffic spikes when export volume peaks.
• The backend/vendor dashboard shows ingest at or above the account rate limit.
• otelcol_exporter_send_failed_spans climbs while the backend itself is healthy (not 5xx).

Common Root Causes

• Account or tier rate limit — the vendor caps requests-per-second or spans-per-minute and you have crossed it.
• No backoff on retries — retries fire immediately, hammering the limiter and extending the throttle window.
• Ignored Retry-After — the exporter retries before the server-advised interval elapses, so the 429s persist.
• Unsampled firehose — 100% of spans are exported when a fraction would satisfy observability needs.
• Undersized batching — many tiny requests burn the request-rate quota faster than fewer, larger batches would.
• Fan-out from many Collectors — several agents export to one backend with no shared rate awareness.

Diagnostic Workflow

Confirm the backend is throttling (429) rather than failing (5xx), and capture any Retry-After guidance:

journalctl -u otelcol-contrib --since '15 min ago' | grep -i '429\|too many requests\|retry-after'
curl -s http://localhost:8888/metrics | grep -E 'otelcol_exporter_send_failed|otelcol_exporter_sent'

Enable retry with exponential backoff so throttled batches are held and re-sent gently instead of amplifying the limit. The otlphttp exporter honors Retry-After automatically when retry_on_failure is enabled:

exporters:
  otlphttp:
    endpoint: https://otel.example.com
    compression: gzip
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 60s        # cap the backoff step
      max_elapsed_time: 600s   # keep retrying through a long throttle window
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 10000        # buffer while backoff waits out the limit

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024      # fewer, fuller requests use less request-rate quota
    send_batch_max_size: 2048

Reduce the volume you send with head sampling so you stay under the quota. The probabilistic_sampler processor keeps a deterministic percentage of traces:

processors:
  probabilistic_sampler:
    sampling_percentage: 25    # export 25% of traces to fit the rate limit
  filter/drop_health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'   # drop noisy health checks

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/drop_health, probabilistic_sampler, batch]
      exporters: [otlphttp]

Example Root Cause Analysis

After a new service rolled out, a Collector began logging HTTP Status Code 429, Message=Too Many Requests every few seconds. The vendor plan allowed 1,000 requests/minute, but the Collector exported unsampled traces with send_batch_size: 128, producing far more, smaller requests than the quota permitted. Because retry_on_failure was disabled, each 429 was immediately resent, which kept the account pinned in the throttle window.

The fix had two parts. First, retry_on_failure was enabled with initial_interval: 5s and max_interval: 60s, so throttled batches backed off and honored the server’s Retry-After instead of hammering the limiter. Second, a probabilistic_sampler at sampling_percentage: 25 plus a larger send_batch_size: 1024 cut request volume roughly 8x, dropping steady-state usage well under the 1,000/minute ceiling. The 429s cleared within a minute and export throughput stabilized.

Prevention Best Practices

• Always enable retry_on_failure with a bounded exponential backoff so throttling is absorbed, not amplified.
• Right-size batches (send_batch_size) so you send fewer, fuller requests and spend less request-rate quota.
• Apply head sampling (probabilistic_sampler) or tail sampling to keep steady-state volume comfortably under your plan.
• Drop low-value telemetry (health checks, synthetic probes) with the filter processor before it reaches the exporter.
• Alert on sustained otelcol_exporter_send_failed_spans and correlate with the vendor’s rate-limit dashboard.
• Know your quota and headroom; capacity-plan sampling rates before a launch rather than after the 429s start.

Quick Command Reference

# Watch for throttling and Retry-After hints
journalctl -u otelcol-contrib -f | grep -i '429\|too many\|retry-after'

# Compare sent vs failed to gauge the throttle rate
curl -s http://localhost:8888/metrics | grep -E 'otelcol_exporter_sent|send_failed'

# Validate config after adding sampling/retry
otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

# Confirm the sampler is in the pipeline
curl -s http://localhost:8888/metrics | grep otelcol_processor

Conclusion

A 429 Too Many Requests on OTLP export means the backend is throttling you, not failing. The durable fix pairs a well-behaved client with less traffic: enable retry_on_failure so backoff honors the server’s Retry-After, and cut volume with sampling, filtering, and larger batches so steady-state export stays under your quota. Retrying harder only extends the throttle; sending smarter ends it.

Related

• OpenTelemetry Error Guide: ‘sending queue is full’
• OpenTelemetry Error Guide: ‘no more retries left; dropping data’
• OpenTelemetry Error Guide: ‘context deadline exceeded’