🚨 Telegraf Error Guide: 'outputs.http 429 Too Many Requests' — Fix Rate-Limited Writes

Overview

When Telegraf pushes metrics over HTTP to a rate-limited endpoint (a SaaS metrics API, InfluxDB Cloud, or a gateway), exceeding the quota returns HTTP 429:

E! [outputs.http] When writing to [https://metrics.example.com/api/v1/write]: received status code 429 (Too Many Requests), body: {"error":"rate limit exceeded, retry after 30s"}

The InfluxDB Cloud variant is similar:

E! [outputs.influxdb_v2] Failed to write metric (429 Too Many Requests): rate limited; retry after ...

429 is throttling, not a permanent failure. Telegraf buffers and retries, ideally honoring Retry-After, but sustained over-limit traffic drops metrics once the buffer overflows.

Symptoms

• received status code 429 (Too Many Requests) clustered around flush times.
• The response body includes rate limit exceeded and often a Retry-After hint.
• Failures scale with the number of agents or a rise in metric cardinality/volume.
• Buffer grows during throttling; metric buffer overflow follows if it persists.

Common Root Causes

• Write rate exceeds the plan/quota — too many requests per second or too many series/points per interval.
• Too many small flushes — a short flush_interval produces many requests instead of fewer large ones.
• A synchronized fleet — hundreds of agents flushing on the same boundary create a thundering herd.
• No compression — larger, more frequent payloads counting harder against byte-based limits.
• High cardinality inflating point volume beyond the quota.

Diagnostic Workflow

Read the response body and any Retry-After value from the logs:

journalctl -u telegraf --since '15 min ago' | grep -iE '429|too many|retry-after'

Reduce request frequency and enable compression and jitter in the output:

[[outputs.http]]
  url = "https://metrics.example.com/api/v1/write"
  method = "POST"
  data_format = "influx"
  content_encoding = "gzip"

[agent]
  interval = "10s"
  flush_interval = "30s"
  flush_jitter = "5s"
  metric_batch_size = 5000
  metric_buffer_limit = 100000

Reproduce the limit and observe headers to learn the actual quota:

for i in $(seq 1 50); do
  curl -o /dev/null -s -w '%{http_code} ' \
    -XPOST "https://metrics.example.com/api/v1/write" \
    -H "Authorization: Bearer $TOKEN" --data-binary 'probe value=1'
done; echo

Inspect rate-limit headers if the API exposes them:

curl -i -XPOST "https://metrics.example.com/api/v1/write" \
  -H "Authorization: Bearer $TOKEN" --data-binary 'probe value=1' \
  | grep -i 'x-ratelimit\|retry-after'

Fewer, larger, compressed requests spread by jitter almost always bring a fleet back under quota; if volume is still too high, aggregate before sending.

Example Root Cause Analysis

A 300-agent fleet pushing to InfluxDB Cloud began seeing 429s every minute. Every agent used the default flush_interval = "10s" with no jitter, so all 300 fired within the same second — a burst that blew the per-second request quota even though average volume was fine. Adding flush_jitter = "10s" and lengthening flush_interval to 30s spread the requests across the window and cut request count 3x; enabling content_encoding = "gzip" reduced bytes. 429s disappeared without a plan upgrade. For the highest-volume clusters they also added [[aggregators.basicstats]] to pre-roll per-minute stats, cutting point volume further.

Prevention Best Practices

• Add flush_jitter so a fleet never synchronizes into a thundering herd against a rate limit.
• Prefer fewer, larger requests: a longer flush_interval with a bigger metric_batch_size beats many small flushes.
• Enable content_encoding = "gzip" to stay under byte-based quotas.
• Aggregate or downsample high-volume, high-cardinality metrics with [[aggregators.*]] before shipping.
• Size metric_buffer_limit to ride out Retry-After backoff windows without dropping.
• Know your provider’s quota and alert on 429 rate so you scale the plan before hitting sustained loss.

Quick Command Reference

# Read 429 bodies and Retry-After
journalctl -u telegraf --since '15 min ago' | grep -iE '429|retry-after'

# Reproduce throttling
for i in $(seq 1 50); do curl -o /dev/null -s -w '%{http_code} ' \
  -XPOST "https://metrics.example.com/api/v1/write" \
  -H "Authorization: Bearer $TOKEN" --data-binary 'probe value=1'; done; echo

# Inspect rate-limit headers
curl -i -XPOST "https://metrics.example.com/api/v1/write" \
  -H "Authorization: Bearer $TOKEN" --data-binary 'probe value=1' \
  | grep -i 'x-ratelimit\|retry-after'

# Watch for the error live
journalctl -u telegraf -f | grep 429

Conclusion

outputs.http 429 Too Many Requests means you are exceeding the endpoint’s rate limit — usually too many small, synchronized flushes rather than raw data volume. Read the Retry-After, then reduce request frequency with a longer flush_interval, add flush_jitter to de-synchronize a fleet, enable gzip, and aggregate high-volume metrics. Size the buffer to absorb backoff so throttling never becomes data loss. If the buffer fills during sustained throttling, see the metric buffer overflow guide. More fixes in the Telegraf guides.