🚨 Telegraf Error Guide: 'metric buffer overflow' — Fix Dropped Metrics on a Full Buffer

Overview

Telegraf keeps a per-output in-memory buffer of metrics waiting to be written. When an output cannot flush fast enough, that buffer fills and the oldest metrics are discarded:

W! [outputs.influxdb] Metric buffer overflow; 1200 metrics have been dropped

This is a symptom, not a root cause. The buffer overflows because something downstream is failing or slow: the output is erroring, the destination is unreachable, or the ingest rate exceeds what the output can flush per interval.

Symptoms

• Repeated Metric buffer overflow; N metrics have been dropped warnings.
• Gaps in dashboards that correspond to the overflow windows.
• The overflow warning is usually accompanied by an output error (connection refused, timeout, 401, partial write).
• internal plugin metrics show buffer_size pinned at buffer_limit.

Common Root Causes

• The output is failing — auth errors, 404, TLS failures, or a down InfluxDB mean nothing flushes and the buffer fills.
• The destination is slow — high write latency so each flush drains less than one interval’s worth of new metrics.
• metric_buffer_limit too small for the collection rate during a transient outage.
• flush_interval too long relative to interval, letting metrics accumulate between flushes.
• A single slow output blocking a shared buffer when multiple outputs are configured without isolation.
• Ingest spikes (a new host fleet, a high-cardinality plugin) exceeding steady-state flush throughput.

Diagnostic Workflow

Find the real error next to the overflow warning — the overflow alone never tells you why:

journalctl -u telegraf --since '15 min ago' | grep -iE 'overflow|error|refused|timeout|401|404'

Enable the internal plugin so you can graph buffer pressure and write latency:

[[inputs.internal]]
  collect_memstats = true

[agent]
  interval = "10s"
  flush_interval = "10s"
  metric_buffer_limit = 100000
  metric_batch_size = 5000

Inspect the current buffer usage and write timings:

telegraf --config /etc/telegraf/telegraf.conf --test --input-filter internal \
  | grep -E 'buffer_size|buffer_limit|write_time_ns|metrics_dropped'

Confirm the destination is actually reachable and fast enough:

curl -o /dev/null -s -w 'http=%{http_code} time=%{time_total}s\n' \
  -XPOST "http://influxdb:8086/write?db=telemetry" --data-binary 'probe value=1'

If http=204 with a low time_total, the destination is healthy and the issue is buffer sizing or batch tuning; if it errors or is slow, fix the output first — enlarging the buffer only delays the drops.

Example Root Cause Analysis

A monitoring host began logging Metric buffer overflow every few minutes. The overflow warning sat directly beneath E! [outputs.influxdb] ... i/o timeout. curl to InfluxDB took 9 seconds per write — the InfluxDB server was CPU-saturated from an unbounded high-cardinality tag. The buffer was a red herring: increasing metric_buffer_limit from 10k to 100k only pushed the overflow out by a few minutes. The real fix was upstream — a processors.tag_limit plus dropping a per-request UUID tag cut cardinality, InfluxDB write latency fell to 40ms, the buffer drained, and overflows stopped. Buffer sizing was then raised modestly to absorb future transient blips.

Prevention Best Practices

• Treat overflow as an alert on output health, not a tuning knob; always find and fix the accompanying write error first.
• Enable [[inputs.internal]] and alert on buffer_size / buffer_limit > 0.8 and on metrics_dropped > 0.
• Size metric_buffer_limit to cover your worst tolerable outage: rate_per_sec × outage_seconds, then add headroom.
• Keep metric_batch_size large enough to drain the buffer faster than it fills at peak ingest.
• For outages longer than memory can hold, add a disk-backed queue (e.g. an outputs.file spool or a message-queue output) so nothing is lost.
• Control cardinality at the source with processors.tag_limit / processors.dedup to keep write latency low.

Quick Command Reference

# Find the real error behind the overflow
journalctl -u telegraf --since '15 min ago' | grep -iE 'overflow|error|timeout'

# Inspect buffer pressure via the internal plugin
telegraf --config /etc/telegraf/telegraf.conf --test --input-filter internal \
  | grep -E 'buffer_size|metrics_dropped|write_time_ns'

# Measure destination write latency
curl -o /dev/null -s -w 'http=%{http_code} time=%{time_total}s\n' \
  -XPOST "http://influxdb:8086/write?db=telemetry" --data-binary 'probe value=1'

# Watch overflow events live
journalctl -u telegraf -f | grep 'buffer overflow'

Conclusion

metric buffer overflow never fires on its own — it means an output stopped keeping up, so the buffer filled and dropped the oldest metrics. Read the error printed alongside it, confirm the destination is reachable and fast with curl, and fix that root cause before touching buffer sizes. Then size metric_buffer_limit and metric_batch_size to survive your expected outage window, and add disk buffering for anything longer. For the connection failures that commonly trigger overflow, see the could not write guide. More fixes in the Telegraf guides.