🚨 Prometheus Error Guide: 'out of bounds' Sample Too Old or Too Far in the Future

Exact Error Message

out of bounds is a TSDB append-path rejection. It appears in the Prometheus log alongside the generic ingestion warning:

ts=2026-06-27T09:12:44.501Z caller=scrape.go:1654 level=warn component="scrape manager" scrape_pool=batch msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=44

The underlying TSDB error string is err-out-of-bounds / out of bounds, and it is returned to remote-write clients in the HTTP body:

err="out of bounds"

user=... err="out of bounds: 1718000000000 < minValidTime 1718450000000"

The scrape or remote-write succeeds at the HTTP level; the individual samples are dropped, so data silently goes missing.

What the Error Means

The TSDB head block has a minimum valid time (minValidTime) — the oldest timestamp it will accept into the in-memory head. Any sample whose timestamp is older than that boundary is rejected as out of bounds. The boundary advances as the head is truncated and older data is compacted into immutable persistent blocks; you cannot append into a block that has already been written.

out of bounds therefore means one of two things:

• The sample is too old — its timestamp is before the head’s minValidTime (already-compacted territory), or
• The sample is too far into the future — its timestamp is beyond the accepted future tolerance, almost always from a target whose clock is ahead.

This is distinct from out of order sample, which is about a sample being older than the last sample of the same series but still inside the head window, and from duplicate sample for timestamp, which is about a same-timestamp, different-value collision. out of bounds is specifically about crossing the head’s absolute time boundary.

Common Causes

• Clock skew on a target. A node whose clock is minutes ahead emits future-stamped samples (when it sets its own timestamps) that exceed the future tolerance; a clock far behind produces too-old samples.
• Remote-writing historical data. Pushing old samples from an edge agent, a backfill job, or a replayed buffer through remote_write — the receiver’s head has already moved past those timestamps.
• Backfilling without the right tooling. Trying to ingest old data via the normal append path instead of promtool tsdb create-blocks-from.
• Out-of-order window not enabled. Legitimately late samples (edge agents, slow networks) arriving beyond the head boundary are rejected when out_of_order_time_window is 0s.
• Exporters exposing stale explicit timestamps. Batch/push exporters that emit an absolute (13-digit ms) timestamp that is older than the head boundary.
• A long Prometheus outage. After a multi-hour gap, agents replaying buffered samples may target timestamps already past the new minValidTime.

How to Reproduce the Error

Remote-write a sample with a deliberately old timestamp to a fresh Prometheus (with the remote-write receiver enabled):

# A timestamp 30 days in the past, in ms
OLD_TS=$(( ($(date +%s) - 30*24*3600) * 1000 ))
cat > /tmp/old.txt <<EOF
test_metric 1 ${OLD_TS}
EOF
# Push via the remote-write/import path; the receiver rejects it:
journalctl -u prometheus -f | grep -i 'out of bounds'

level=warn msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1

A target with a fast clock reproduces the future case: set a node’s clock 10 minutes ahead, have it expose a metric with its own timestamp, and Prometheus rejects the future sample.

Diagnostic Commands

Check the head’s time bounds — anything older than minTime will be rejected:

curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.headStats'

{ "numSeries": 184221, "minTime": 1718450000000, "maxTime": 1718536400000, ... }

Watch the rejection counters by reason:

rate(prometheus_tsdb_out_of_bound_samples_total[5m])

increase(prometheus_target_scrapes_sample_out_of_bounds_total[10m])

Compare a target’s clock against the Prometheus host (the most common root cause):

chronyc tracking | grep -E 'System time|Last offset'
date -u

System time     : 612.004 seconds fast of NTP time

Inspect a suspect target for explicit (trailing 13-digit ms) timestamps:

curl -s http://10.0.7.4:9100/metrics | grep -E ' [0-9]{13}$' | head

List persistent blocks to see how far back the data already is (you cannot append before the newest block boundary that has been compacted):

promtool tsdb list /var/lib/prometheus/data
du -sh /var/lib/prometheus/data/*

Step-by-Step Resolution

1. Fix the clock first. If chronyc tracking shows a multi-second-or-worse offset on the target or on Prometheus, correct NTP. Future-sample rejections almost always vanish once clocks are sane:

sudo chronyc makestep
systemctl status chronyd

2. Enable an out-of-order window for legitimately late data. If edge agents or HA remote-write produce samples that are merely late (not ancient), accept them within a bounded window:

storage:
  tsdb:
    out_of_order_time_window: 30m

Reload (curl -X POST http://localhost:9090/-/reload). Samples up to 30 m late are now ingested instead of rejected.

3. Do not remote-write historical data — backfill it. For genuinely old data (hours/days), bypass the append path and create blocks offline, then move them into the data dir:

promtool tsdb create-blocks-from openmetrics /tmp/backfill.om /var/lib/prometheus/data
systemctl reload prometheus   # or restart; blocks are picked up on compaction

The OpenMetrics input file must carry explicit timestamps; create-blocks-from writes proper persistent blocks regardless of the head boundary.

4. Stop exporters emitting stale explicit timestamps. Let Prometheus assign the scrape timestamp unless the workload is a true batch job, and ensure any explicit timestamp always advances.

5. After an outage, prefer letting agents drop their oldest buffered samples (or backfill them) rather than replaying them through remote-write into a head that has moved on.

Prevention and Best Practices

• Run NTP/chrony everywhere and alert on offsets over ~1 s; future-sample out of bounds is almost entirely a clock problem.
• Treat remote_write as a path for fresh data only. Use promtool tsdb create-blocks-from for any historical import.
• Set a deliberate out_of_order_time_window sized to your worst-case agent lateness (e.g. 30 m), not larger than needed.
• Alert on prometheus_tsdb_out_of_bound_samples_total and prometheus_target_scrapes_sample_out_of_bounds_total so silent drops surface.
• Keep --storage.tsdb.retention.time long enough that backfills land inside retention, or the blocks will be deleted on the next compaction.

Related Errors

• out of order sample — sample older than the previous one for the same series, but still inside the head window. Different boundary, different fix.
• duplicate sample for timestamp — same timestamp, different value, from two sources producing one series.
• too old or too far into the future — the human-readable wrapper logged around out of bounds; same root causes.

Frequently Asked Questions

Is out of bounds the same as out of order sample? No. out of order sample means a sample is older than the last stored sample of that series but still within the head’s accepted window. out of bounds means the timestamp is outside the head’s absolute min/max valid time — usually too old (already compacted) or too far in the future (clock skew).

Will enabling out_of_order_time_window fix future-stamped samples? No. The out-of-order window only accepts late (past) samples up to the window size. Samples in the future are still rejected — fix the target’s clock instead.

How do I ingest a week of old data without this error? Use promtool tsdb create-blocks-from. It builds persistent blocks directly and is not subject to the head’s minValidTime. Remote-writing or scraping that data will always be rejected as out of bounds.

Why does only one target trigger this? That target almost certainly has clock skew or emits its own (stale or future) explicit timestamps. Compare chronyc tracking on it against the Prometheus host and grep its /metrics for trailing 13-digit timestamps.