🚨 Loki Error Guide: 'timestamp too new' — Fix Clock Skew Before the Distributor Rejects Future-Dated Logs

Overview

Loki’s distributor validates the timestamp of every log entry as it ingests a push. If an entry is dated too far into the future, the distributor rejects it with an HTTP 400 and a message like this:

entry for stream '{app="api", namespace="prod"}' has timestamp too new: 2026-07-12T18:04:11Z

The boundary is limits_config.creation_grace_period, which defaults to 10m. Any entry whose timestamp is later than now + creation_grace_period is rejected, and the drop is counted as loki_discarded_samples_total{reason="too_far_in_future"}. This is the mirror image of the old-timestamp path: entries dated too far in the past are handled by reject_old_samples and surface as entry out of order or old-sample rejections, which is a separate problem with a separate fix. A future-dated timestamp almost always means the sender’s clock is wrong or your timestamp-parsing stage is misreading the field — not that Loki is misbehaving.

Symptoms

• Pushes fail with HTTP 400 and timestamp too new in the distributor logs.
• loki_discarded_samples_total{reason="too_far_in_future"} increases, often from one host or one deployment.
• Logs from a specific node or pod go missing while other sources ingest normally.
• The rejected timestamps are minutes or hours ahead of wall-clock time, sometimes exactly one timezone offset ahead.
• The problem appears right after a node reboot, a VM migration, or a change to the agent’s timestamp stage.

Common Root Causes

• Clock skew on the log source — NTP is not running or has drifted, so the host’s clock is ahead of real time and every entry looks future-dated.
• Wrong timezone parsing — the timestamp stage parses a local time as UTC (or applies the wrong location), shifting entries forward by the timezone offset.
• Bad timestamp format string — a mismatched format in Promtail/Alloy misreads day/month or hour fields and produces a future date.
• Application emitting future timestamps — a service with a misconfigured clock or a deliberately post-dated event field that the pipeline trusts.
• VM/container clock jumps — a paused or migrated VM resumes with a clock ahead of the host, briefly emitting future-dated logs.
• Grace period too tight for real skew — a legitimately distributed fleet with small, bounded skew that occasionally exceeds a very small creation_grace_period.

How to diagnose

1. Confirm the rejection reason and read the offending timestamps in the distributor logs:

   kubectl logs -l app=loki,component=distributor -n loki --tail=300 \
     | grep 'timestamp too new'

2. Quantify the discards to see whether it is one source or a fleet-wide issue:

   sum by (reason) (
     rate(loki_discarded_samples_total{reason="too_far_in_future"}[5m])
   )

3. Check clock sync on the suspected sender — compare the node clock against a trusted source:

   kubectl debug node/my-cluster-node-3 -it --image=busybox -- date -u
   chronyc tracking   # on the host: look at System time offset

4. Read the effective grace period so you know the exact future boundary:

   limits_config:
     creation_grace_period: 10m
     reject_old_samples: true
     reject_old_samples_max_age: 168h

5. Inspect the timestamp stage in the agent for a format or timezone mistake:

   pipeline_stages:
     - timestamp:
         source: ts
         format: RFC3339
         location: America/New_York   # wrong location shifts entries forward

Fixes

Fix time sync on the senders first — ensure NTP/chrony is running and healthy on every log source, since a corrected clock removes the rejections at the root:

sudo systemctl enable --now chronyd
chronyc makestep          # step the clock immediately
chronyc sources -v        # confirm a reachable, selected source

Correct the timestamp parsing stage — set the right format and location so the parsed time matches the event’s real instant:

pipeline_stages:
  - timestamp:
      source: ts
      format: '2006-01-02T15:04:05.000Z07:00'
      location: UTC

Fall back to ingestion time when the source timestamp is untrustworthy — drop the custom timestamp stage so Loki stamps entries at receipt, avoiding future-dating entirely:

pipeline_stages:
  - json:
      expressions:
        level: level
  # no 'timestamp:' stage — Loki uses receive time

Raise creation_grace_period only for legitimate, bounded skew — if a distributed fleet has small real skew you cannot eliminate, widen the window deliberately rather than losing logs:

limits_config:
  creation_grace_period: 30m   # only if the future skew is genuine

What to watch out for

• too_far_in_future and old-sample/out-of-order rejections are opposite problems; raising creation_grace_period does nothing for past-dated entries governed by reject_old_samples.
• A large creation_grace_period widens the window in which genuinely bad future timestamps are accepted, which can distort dashboards and alert evaluation — keep it as tight as your real skew allows.
• Fixing the clock stops new rejections but does not recover the logs already dropped; those pushes are gone.
• A timezone bug often shows up as a rejection offset exactly equal to the zone’s UTC offset — that pattern points straight at the timestamp stage, not NTP.
• Watch for a single noisy node dominating too_far_in_future; fleet-wide skew and one broken clock need very different responses.

Related

• Loki Error Guide: ‘entry out of order’ — the past-timestamp counterpart governed by reject_old_samples.
• Loki Error Guide: ‘Ingestion rate limit exceeded’ — another distributor-side push rejection to distinguish from timestamp drops.
• Loki Error Guide: ‘Per stream rate limit exceeded’ — the per-stream throughput limit that can co-occur during a bad backfill.