🚨 Loki Error Guide: 'RequestTimeTooSkewed status code: 403' — Fix Clock Drift, Not IAM

Overview

Loki logs this error when the timestamp it signs into an object-store request differs too far from the store’s own clock, and the store rejects the signature with a 403:

RequestTimeTooSkewed: The difference between the request time and the current time is too large.
	status code: 403, request id=..., host id=...

Ingesters emit it on flush, the compactor on compaction, and queriers on reads. Despite the 403 status, this is not a permissions problem: S3 (and S3-compatible stores) allow only a small window — typically 15 minutes — between the signed request time and server time, and Loki’s host clock has drifted outside it. The fix is time synchronization on the node or pod, not IAM.

Symptoms

• Ingesters log RequestTimeTooSkewed ... status code: 403 on every flush even though IAM is unchanged.
• The failures are cluster-wide or node-local, tracking specific nodes whose clocks drifted.
• The error appeared after a VM was paused/resumed, a node came back from suspend, or NTP stopped.
• date -u inside the pod differs noticeably from real UTC.
• AccessDenied never appears — only RequestTimeTooSkewed, distinguishing it from a genuine policy 403.

Common Root Causes

• Node clock drift — the underlying host clock has wandered outside the store’s allowed skew window.
• Missing or broken NTP/chrony — the time daemon is not installed, not running, or cannot reach its time sources.
• Paused or migrated VM — a suspended/resumed or live-migrated VM resumes with a stale clock until it resyncs.
• Container clock skew — the pod inherits a bad host clock, or a virtualization layer is not passing accurate time through.
• Firewalled NTP — outbound UDP/123 is blocked, so chrony silently never disciplines the clock.

How to diagnose

1. Confirm the error is skew, not permissions — grep the logs and check that it is RequestTimeTooSkewed, not AccessDenied:

   kubectl logs -l app=loki,component=ingester --since=10m \
     | grep -iE 'RequestTimeTooSkewed|status code: 403'

2. Compare the pod clock to real UTC from inside the container:

   kubectl exec -it deploy/loki-ingester -- date -u
   # compare against a known-good UTC source

3. Check the node’s time synchronization state on the host running the pod:

   kubectl get pod -l app=loki,component=ingester -o wide   # find the node
   # then on that node:
   chronyc tracking     # or: timedatectl status

4. Confirm NTP reachability so you know the daemon can actually discipline the clock:

   chronyc sources -v
   # look for a reachable, selected source (^*)

5. Isolate skew to specific nodes by checking clock offset across the fleet, since the failure is usually node-local:

   # example alerting expression on node clock offset
   - alert: NodeClockSkew
     expr: abs(node_timex_offset_seconds) > 30
     for: 5m

Fixes

Sync the node clock with chrony or NTP so it stays inside the store’s allowed window. Install and enable chrony on the node, point it at reliable sources, and confirm it disciplines the clock:

# on the affected node
sudo apt-get install -y chrony
sudo systemctl enable --now chrony
chronyc makestep      # step the clock immediately once
chronyc tracking      # confirm offset is now small

Ensure the kubelet node time is correct by fixing the underlying host, since pods inherit the host clock. Verify with a one-off job that the in-cluster time is now sane:

apiVersion: batch/v1
kind: Job
metadata:
  name: clock-check
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: clock
          image: busybox
          command: ["sh", "-c", "date -u; sleep 1"]

Treat the 403 as a clock issue, not IAM — do not touch the S3 policy or credentials for RequestTimeTooSkewed. Confirm the store accepts requests again once time is synced:

kubectl exec -it deploy/loki-ingester -- sh -c \
  'date -u && aws s3 ls s3://loki-chunks-prod/'

Monitor node clock offset so drift is caught before it breaks flushes. Alert on node_timex_offset_seconds and on the sync status so a stopped chrony surfaces early:

- alert: LokiRequestTimeTooSkewed
  expr: abs(node_timex_offset_seconds) > 60
  for: 2m
  labels: { severity: warning }
  annotations:
    summary: "Node clock drift may cause Loki RequestTimeTooSkewed 403s"

What to watch out for

• The 403 status is misleading: RequestTimeTooSkewed is a clock problem, and changing IAM policy or rotating credentials will not fix it.
• Skew is usually node-local, so a subset of ingesters can fail while others flush fine; check per-node, not just cluster-wide.
• Paused, snapshotted, or live-migrated VMs frequently resume with a stale clock until chrony steps it; chronyc makestep forces an immediate correction.
• If NTP/UDP 123 is firewalled, chrony runs but never disciplines the clock; confirm a source is actually reachable and selected.
• The allowed window is small (about 15 minutes on S3); even modest sustained drift will eventually cross it, so alert on offset well below that threshold.

Related

• Loki Error Guide: ‘AccessDenied: Access Denied status code: 403’ — the genuine-permissions 403 to rule out before blaming the clock.
• Loki Error Guide: ‘failed to flush chunks’ — the flush-path symptom a persistent skew 403 produces.
• Loki Error Guide: ‘entry out of order’ — another timestamp-driven failure, this time on the ingestion path.