🚨 Loki Error Guide: 'entry out of order' / 'entry too far behind' — Fix Timestamp Ordering at Ingest

Overview

Loki’s ingester rejects log lines whose timestamps break the ordering rules for a stream. The classic out-of-order message (Loki before out-of-order ingestion was enabled by default) is:

entry out of order for stream: {app="api"}; last timestamp: 2026-07-09T10:15:04Z, incoming timestamp: 2026-07-09T10:15:01Z

The related “too far behind” variant, which persists even with unordered writes allowed, is:

entry too far behind, entry timestamp is: 2026-07-09T09:00:00Z, oldest acceptable timestamp is: 2026-07-09T10:00:00Z

Within a single stream Loki tracks the most recent accepted timestamp. entry out of order means an incoming line is older than the last accepted line for that stream. entry too far behind means a line is older than the accepted window (max_chunk_age back from the head) even when unordered writes are permitted. Both drop the offending lines at ingest, so the real cost is missing logs.

Symptoms

• Promtail/Alloy logs show repeated entry out of order or entry too far behind with 400-class push errors.
• Some log lines never appear in Grafana even though the source clearly emitted them.
• The error names two timestamps that are close together (seconds apart) — a sign of interleaving, not a huge clock jump.
• loki_discarded_samples_total with reason out_of_order or too_far_behind climbs.
• Problems concentrate on hosts with clock skew or on multi-writer streams.

Common Root Causes

• Multiple writers to one stream — several pods/threads share identical labels, so their interleaved timestamps go backwards relative to each other.
• Clock skew — a source host’s clock is behind, so its lines look old.
• Backfilling old logs past the accepted window triggers too far behind.
• unordered_writes disabled on older configs, making any regression fatal.
• Batching/buffering in the client that reorders lines before sending.
• max_chunk_age too small, shrinking the window of acceptable timestamps.

Diagnostic Workflow

Confirm the discard reason on the distributor/ingester metrics:

curl -s http://loki-ingester:3100/metrics \
  | grep -E 'loki_discarded_samples_total.*(out_of_order|too_far_behind)'

Check the ordering and window settings actually in force:

limits_config:
  unordered_writes: true            # allow out-of-order within the window
  reject_old_samples: true
  reject_old_samples_max_age: 168h  # lines older than this are rejected outright

ingester:
  max_chunk_age: 2h                 # defines how far "behind" is acceptable

Look for clock skew across log sources:

# On each suspect host
timedatectl status | grep -E 'synchronized|NTP'
chronyc tracking 2>/dev/null | grep 'System time'

If two writers share labels, confirm cardinality: a stream that should be per-instance but lacks the instance/pod label will interleave timestamps and trip out-of-order even with correct clocks.

Example Root Cause Analysis

A fleet of workers all logged under {job="batch"} with no per-host label. Three hosts wrote into the same Loki stream. Host A’s clock was 3 seconds ahead of Host B. When B’s line (timestamped 10:15:01) arrived after A’s line (10:15:04) had already been accepted for the shared stream, Loki saw the timestamp go backwards and dropped B’s line with entry out of order.

The fix had two parts. First, they added a host label in the scrape config so each host owned its own stream — eliminating cross-writer interleaving entirely. Second, they enforced NTP with chrony across the fleet to remove the 3-second skew. A separate nightly job that backfilled day-old logs was tripping entry too far behind; they raised reject_old_samples_max_age to 168h so legitimate backfill inside a week was accepted. Discards dropped to zero.

Prevention Best Practices

• Give each writer its own stream via a bounded label (host, pod, instance) so timestamps never interleave.
• Keep unordered_writes: true (the modern default) so brief regressions inside the window are tolerated.
• Enforce NTP/chrony on every log source to eliminate clock skew.
• Size reject_old_samples_max_age and max_chunk_age to cover legitimate backfill windows.
• Have clients send in timestamp order and avoid reordering buffers.
• Alert on loki_discarded_samples_total{reason=~"out_of_order|too_far_behind"} to catch silent log loss.

Quick Command Reference

# Discards by ordering reason
curl -s localhost:3100/metrics | grep -E 'out_of_order|too_far_behind'

# Verify clock sync on a source host
timedatectl status | grep synchronized

# Confirm ordering settings
curl -s http://loki:3100/config | grep -E 'unordered_writes|reject_old_samples|max_chunk_age'

limits_config:
  unordered_writes: true
  reject_old_samples_max_age: 168h
ingester:
  max_chunk_age: 2h

Conclusion

entry out of order and entry too far behind both mean Loki dropped log lines because their timestamps violated a stream’s ordering or age window. The durable fix is structural: give each writer its own stream with a bounded label so timestamps stop interleaving, keep clocks in sync with NTP, and size the age/chunk windows to cover legitimate backfill. With unordered_writes enabled and discard alerts in place, you catch and prevent the silent log loss these errors cause.