🚨 Telegraf Error Guide: 'partial write: points beyond retention policy' — Fix Timestamps

Overview

InfluxDB drops points whose timestamps fall outside the target retention policy’s window, and Telegraf reports it as a partial write:

E! [outputs.influxdb] When writing to [http://influxdb:8086]: received error status code 400; partial write: points beyond retention policy dropped=42

As with other partial writes, in-window points are accepted and only the out-of-range ones are dropped. The rejected points are almost always too old for the retention policy — or, when precision is misconfigured, absurdly far in the future.

Symptoms

• partial write: points beyond retention policy dropped=N on flushes that include historical or backfilled data.
• Real-time metrics write fine; only backfill, replay, or a specific lagging source is dropped.
• After a precision misconfiguration, timestamps land in the year 55000+ or 1970 and get dropped.
• Reducing a retention policy’s duration suddenly starts dropping previously-fine points.

Common Root Causes

• Timestamps older than the retention policy duration — backfilling 90 days into a 7-day RP.
• Wrong precision in the output plugin (e.g. s vs ns) making InfluxDB interpret the epoch as seconds-since-1970 as if nanoseconds, throwing timestamps wildly off.
• Clock skew on the collecting host pushing timestamps into the future beyond the RP’s future tolerance.
• A shortened retention policy where the new duration excludes data still in Telegraf’s buffer.
• Replaying an archived queue of metrics whose timestamps are now stale.

Diagnostic Workflow

Check the target retention policy’s duration first:

influx -database telemetry -execute 'SHOW RETENTION POLICIES ON telemetry'

name    duration  shardGroupDuration  replicaN  default
----    --------  ------------------  --------  -------
autogen 168h0m0s  24h0m0s             1         true

A 168h (7-day) RP will reject anything older than 7 days. Confirm the output plugin’s precision matches how your inputs stamp time:

[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  database = "telemetry"
  precision = "s"

Reproduce with an obviously old timestamp to confirm the window, then an in-window one:

# Far in the past -> dropped
curl -i -XPOST "http://influxdb:8086/write?db=telemetry&precision=s" \
  --data-binary 'probe value=1 1000000000'

# Now -> accepted
curl -i -XPOST "http://influxdb:8086/write?db=telemetry&precision=s" \
  --data-binary "probe value=1 $(date +%s)"

Check host clock skew, a frequent cause of future-dated drops:

timedatectl status | grep -i 'synchronized\|NTP'
chronyc tracking 2>/dev/null | grep -i 'System time'

Example Root Cause Analysis

A team enabled a disk-buffered queue in front of Telegraf and, after an outage, replayed six hours of metrics into a database whose autogen policy was only 1 hour. Everything older than one hour was dropped with points beyond retention policy. SHOW RETENTION POLICIES confirmed the 1h duration. Rather than lose the replayed data, they created a dedicated backfill retention policy with a 30-day duration and pointed the replay Telegraf at it via retention_policy = "backfill". Live collection kept using autogen; the historical replay landed in backfill. No points were dropped on the second run.

Prevention Best Practices

• Set the output plugin precision to match your inputs and keep it explicit; a mismatched precision is the most common future-dated cause.
• Keep hosts NTP-synchronized and alert on skew so future-dated points never appear.
• Size retention policies to cover your largest expected backfill window, or route backfills to a longer-duration RP.
• Before shortening a retention policy, confirm no buffered or queued data still falls in the range you are cutting.
• Monitor Telegraf’s dropped-metric count so silent retention drops are caught during replays.

Quick Command Reference

# Show retention policy durations
influx -database telemetry -execute 'SHOW RETENTION POLICIES ON telemetry'

# Reproduce an out-of-window vs in-window write
curl -i -XPOST "http://influxdb:8086/write?db=telemetry&precision=s" \
  --data-binary 'probe value=1 1000000000'
curl -i -XPOST "http://influxdb:8086/write?db=telemetry&precision=s" \
  --data-binary "probe value=1 $(date +%s)"

# Check clock sync
timedatectl status | grep -i NTP

# Watch for the error live
journalctl -u telegraf -f | grep 'retention policy'

Conclusion

partial write: points beyond retention policy means timestamps landed outside the RP’s window — usually too old for a backfill, or thrown off by a precision mismatch or clock skew. Read the RP duration with SHOW RETENTION POLICIES, verify precision, and route legitimate historical data to a longer-duration policy instead of dropping it. Keep clocks in NTP sync to eliminate future-dated drops. For type-related partial writes see the field type conflict guide. More fixes in the Telegraf guides.