🚨 VictoriaMetrics Error Guide: 'dropping row with too old timestamp' — Backfill and Retention Fix

Overview

VictoriaMetrics and vmagent log this warning when an incoming sample carries a timestamp that falls outside the window the storage will accept — typically older than the configured retention, or older than the deduplication/scrape interval allows. The row is silently discarded and the data never lands:

2026-07-09T08:14:22.918Z	warn	VictoriaMetrics/lib/storage/table.go:209	dropping row with too old timestamp 1717200000000 (2024-06-01T00:00:00Z); the minimum accepted timestamp is 1746662400000 (2026-05-08T00:00:00Z); see https://docs.victoriametrics.com/#retention

On vminsert in cluster mode the wording is similar, referencing the minimum accepted timestamp derived from -retentionPeriod. The critical consequence: samples matching this condition are dropped, not queued. If you are backfilling historical data, importing from Prometheus, or an agent’s clock is wrong, you can lose large swaths of data while everything appears healthy.

Symptoms

• vmagent or vminsert logs repeat dropping row with too old timestamp ... at warn level.
• A backfill/import job “succeeds” (HTTP 204) but the historical range is absent when you query it.
• Gaps appear at the far-left (oldest) edge of Grafana graphs after an import.
• Data from one specific host or scrape target is missing while others are fine — a clock-skew signature.
• vm_rows_ignored_total increments with reason too_old_timestamp.
• After lowering retention, older-but-still-wanted samples begin getting rejected.

Common Root Causes

• Backfilling outside the retention window — importing data older than -retentionPeriod (default 1 month), so every historical sample is by definition “too old.”
• Retention set too short — -retentionPeriod=1 (one month) rejects anything older than ~31 days even for legitimate late-arriving data.
• Clock skew on a scrape target or agent — an unsynchronized host stamps samples in the past; NTP drift of hours or days pushes them below the minimum accepted timestamp.
• Bulk import via the wrong endpoint — pushing to the normal remote-write path instead of using vmctl, so retention gating applies to the historical timestamps.
• Deduplication interval mismatch — -dedup.minScrapeInterval interacting with out-of-order or delayed samples during replay.
• Timezone or millisecond/second confusion — sending Unix seconds where milliseconds are expected, producing timestamps far in the past.

Diagnostic Workflow

Confirm the log source and read the exact minimum accepted timestamp — it tells you the retention boundary:

sudo journalctl -u vmagent --since '1 hour ago' | grep -i 'too old timestamp'
sudo journalctl -u victoriametrics --since '1 hour ago' | grep -i 'too old timestamp'
# cluster: the message originates on vminsert
sudo journalctl -u vminsert --since '1 hour ago' | grep -i 'too old timestamp'

Check how many rows are being ignored and why, via the self-scraped metrics (single-node 8428, cluster vmstorage 8482):

curl -s http://<node>:8428/metrics | grep -E 'vm_rows_ignored_total|vm_rows_received_total'

Verify the running retention period the process was started with:

sudo journalctl -u victoriametrics | grep -i retentionPeriod
ps -o cmd -C victoria-metrics | tr ' ' '\n' | grep -i retention

Decode the offending timestamp from the log to confirm whether it is a real historical time or a unit mistake:

# The log prints ms; divide by 1000 for date -d @<seconds>
date -d @1717200000    # => Sat Jun  1 00:00:00 UTC 2024

Check for clock skew on the scrape targets. Compare each host’s time to a reference and inspect NTP sync:

timedatectl show -p NTPSynchronized --value
chronyc tracking 2>/dev/null | grep -i 'System time'

Confirm the true minimum accepted timestamp against your retention: if -retentionPeriod=1 (one month) and today is 2026-07-09, anything before ~2026-06-09 is rejected. Any backfill older than that must go through vmctl with an adjusted retention, not the live ingest path.

Example Root Cause Analysis

An SRE team migrated 18 months of history off a decommissioned Prometheus into single-node VictoriaMetrics running with the default -retentionPeriod=1. They ran vmctl prometheus and watched the log fill with dropping row with too old timestamp ...; the minimum accepted timestamp is ... (2026-06-09...). The import reported success, but only the last month of data was queryable.

The root cause was simple and structural: VictoriaMetrics rejects any sample older than its retention window, and one month cannot hold 18 months. vmctl was pushing correct timestamps, but the storage refused everything before the retention boundary.

The fix was to restart VictoriaMetrics with -retentionPeriod=24 (24 months) so the storage would accept the full historical range, then re-run the vmctl prometheus migration. They also added -search.maxStalenessInterval tuning for the gappy legacy data. After the retention change, vm_rows_ignored_total{reason="too_old_timestamp"} stopped incrementing and all 18 months appeared in Grafana. Separately, they discovered one edge collector had a 6-hour clock offset; enabling chrony on it removed a steady trickle of dropped rows unrelated to the migration.

Prevention Best Practices

• Size -retentionPeriod to cover both your live needs and any historical data you intend to import before you start a backfill.
• Always backfill with vmctl (vmctl prometheus, vmctl vm-native, vmctl opentsdb, vmctl influx) rather than replaying old data through the normal remote-write ingest path.
• Keep clocks synchronized everywhere with NTP (chrony/systemd-timesyncd) and alert on timedatectl NTP-desync so no target stamps samples in the past.
• Verify timestamp units before bulk import — VictoriaMetrics expects Unix milliseconds on most ingest paths; sending seconds lands you decades in the past.
• Monitor vm_rows_ignored_total and alert when the too_old_timestamp reason increments, so silent data loss surfaces immediately.
• Set -dedup.minScrapeInterval to match your real scrape interval so deduplication does not interact badly with delayed samples.

Quick Command Reference

# Find the dropped-row warnings and the minimum accepted timestamp
sudo journalctl -u vmagent -u victoriametrics --since '1h' | grep -i 'too old timestamp'

# Count ignored rows and their reason
curl -s http://<node>:8428/metrics | grep vm_rows_ignored_total

# Decode a logged (ms) timestamp
date -d @$((1717200000000/1000))

# Check the running retention
ps -o cmd -C victoria-metrics | tr ' ' '\n' | grep -i retention

# Correct way to backfill historical data (does NOT bypass retention)
vmctl prometheus --prom-snapshot=/path/to/snapshot \
  --vm-addr=http://<node>:8428

# Restart with a longer retention to accept old data
# victoria-metrics -retentionPeriod=24 -storageDataPath=/var/lib/victoria-metrics

# Verify NTP sync on scrape targets
timedatectl show -p NTPSynchronized --value

Conclusion

dropping row with too old timestamp is VictoriaMetrics enforcing its retention and dedup boundaries — samples older than the minimum accepted timestamp are discarded, not buffered, so the risk is silent data loss. The log line hands you the exact cutoff; compare it to your -retentionPeriod and to the timestamps you are sending. If you are importing history, extend retention first and always use vmctl rather than the live ingest path. If only one target is affected, suspect clock skew and fix it with NTP. Watch vm_rows_ignored_total{reason="too_old_timestamp"} so the next occurrence is an alert, not a surprise gap in a dashboard.