🚨 Telegraf Error Guide: 'write failed: 413 Request Entity Too Large' — Fix Batch Size

Overview

The output plugin builds a write request from a batch of metrics. If that request body exceeds a size limit somewhere in the path, the server rejects it:

E! [outputs.influxdb] When writing to [http://influxdb:8086]: received error status code 413 (Request Entity Too Large)

The limit is usually enforced by a reverse proxy (nginx client_max_body_size, an API gateway) or by InfluxDB’s own max-body-size. Because 413 is a non-retryable 4xx, the batch is typically dropped rather than retried.

Symptoms

• received error status code 413 (Request Entity Too Large) on flushes, especially large ones.
• Failures correlate with ingest spikes or a large metric_batch_size.
• Small batches succeed; big ones (backfill, many hosts) fail.
• A proxy access log shows the 413 before the request reaches InfluxDB.

Common Root Causes

• metric_batch_size too large, producing a request body bigger than the server accepts.
• A reverse proxy body limit — nginx default client_max_body_size 1m in front of InfluxDB.
• InfluxDB max-body-size set lower than the batch payload (default 25MB, sometimes tightened).
• Uncompressed writes — not setting content_encoding = "gzip" so bodies are several times larger than they need to be.
• A backfill or replay producing unusually large batches that exceed the steady-state limit.

Diagnostic Workflow

Confirm the batch settings and enable compression in the output plugin:

[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  database = "telemetry"
  metric_batch_size = 1000
  content_encoding = "gzip"

Estimate the payload size at your current batch size — line protocol is roughly 60–120 bytes/point:

telegraf --config /etc/telegraf/telegraf.conf --test --input-filter cpu \
  | wc -c

Find where the 413 originates by probing InfluxDB directly vs through the proxy with a large body:

# Generate ~2MB of line protocol and post it
python3 -c 'print("\n".join(f"probe,i={i} v={i}" for i in range(30000)))' > /tmp/big.lp

curl -i -XPOST "http://influxdb:8086/write?db=telemetry" \
  --data-binary @/tmp/big.lp | head -1        # direct to InfluxDB

curl -i -XPOST "http://influxdb-proxy:80/write?db=telemetry" \
  --data-binary @/tmp/big.lp | head -1        # via the proxy

If the direct call returns 204 but the proxy returns 413, the proxy limit is the culprit; raise it:

# nginx
client_max_body_size 25m;

Example Root Cause Analysis

A backfill job raised metric_batch_size to 50000 to speed ingestion. Writes through the nginx TLS proxy started returning 413 while a direct-to-InfluxDB test of the same payload returned 204 — pinpointing the proxy. nginx still had the default client_max_body_size 1m, and even gzipped, a 50k-point batch exceeded it. Two changes fixed it: content_encoding = "gzip" (already halving the body) and raising the proxy to client_max_body_size 25m to match InfluxDB’s own max-body-size. For steady-state they lowered the batch back to 5000 so normal flushes stayed well under any limit, reserving large batches for the controlled backfill.

Prevention Best Practices

• Set content_encoding = "gzip" on HTTP-based outputs to shrink bodies and reduce the chance of hitting any limit.
• Keep metric_batch_size modest (1000–5000) for steady state; large batches multiply payload size.
• Align limits across the whole path — proxy client_max_body_size, gateway limits, and InfluxDB max-body-size should be consistent and generous.
• Test backfill/replay batch sizes against the real proxy path before running them in bulk.
• Alert on 4xx write errors; because 413 drops the batch, buffer depth will not warn you.

Quick Command Reference

# Show batch + encoding config
telegraf --config /etc/telegraf/telegraf.conf config | grep -A8 outputs.influxdb

# Estimate payload size for a plugin's batch
telegraf --config /etc/telegraf/telegraf.conf --test --input-filter cpu | wc -c

# Compare direct vs proxied with a large body
curl -i -XPOST "http://influxdb:8086/write?db=telemetry" --data-binary @/tmp/big.lp | head -1
curl -i -XPOST "http://influxdb-proxy:80/write?db=telemetry" --data-binary @/tmp/big.lp | head -1

# Watch for the error live
journalctl -u telegraf -f | grep 413

Conclusion

write failed: 413 Request Entity Too Large means the write body exceeded a size limit — usually a reverse proxy’s client_max_body_size, sometimes InfluxDB’s max-body-size, amplified by a big metric_batch_size or uncompressed payloads. Isolate the limit by testing direct vs proxied, enable gzip, keep steady-state batches modest, and align all body limits along the path. Since 413 drops the batch, monitor 4xx write errors. For batches rejected on content rather than size, see the field type conflict guide. More fixes in the Telegraf guides.