🚨 Automation Error: Cron Job Skipped or Ran Twice at a DST Transition

Overview

Twice a year, a job scheduled in a local timezone either does not run at all or runs twice — with no error, no failed exit code, and nothing obviously wrong in the crontab. The symptom shows up in the job’s own logs as a missing or duplicated execution around 02:00 or 03:00 on a daylight-saving-time (DST) transition date:

# Expected nightly billing run at 02:30 America/New_York — SPRING FORWARD (clocks 02:00 -> 03:00)
2026-03-08 01:30:00 EST  billing-run start ... ok
2026-03-09 02:30:00 EDT  <no entry — 02:30 never existed on 2026-03-08>
# The 02:30 run for 2026-03-08 was silently skipped.

# FALL BACK (clocks 02:00 -> 01:00) — the local hour 01:00-02:00 repeats
2026-11-01 01:30:00 EDT  hourly-sync start ... ok
2026-11-01 01:30:00 EST  hourly-sync start ... ok   # same wall-clock label, ran twice

The scheduler did exactly what a local-time cron is defined to do. The bug is that the schedule was expressed in a timezone that has gaps and overlaps, and the job was not idempotent enough to survive them.

Symptoms

• A job that runs every night silently misses exactly one run on the spring-forward date and no error is logged.
• A job runs twice — producing duplicate emails, duplicate charges, or double-counted aggregates — on the fall-back date.
• The anomaly recurs only in March and November (or whenever the local zone changes offset), never in between.
• systemd timers log Persistent catch-up runs at odd times after a DST change.
• Hosts in different timezones disagree about when “the 2 AM job” ran.
• date on the host shows a timezone abbreviation change (EST/EDT, GMT/BST) around the incident.

Common Root Causes

• Schedule expressed in a DST-observing local timezone. A time like 02:30 does not exist on spring-forward and occurs twice on fall-back; classic cron interprets crontab times in the system local zone.
• CRON_TZ / TZ set to a local zone instead of UTC, so Vixie cron and derivatives apply the local offset including DST.
• systemd OnCalendar= with a local timezone. systemd handles the gap/overlap more sanely than Vixie cron but still evaluates in the configured zone; combined with Persistent=true it can fire an extra catch-up run.
• Kubernetes CronJob timezone confusion. Older clusters ignore spec.timeZone and use the controller’s UTC clock; newer ones honor spec.timeZone and reintroduce DST behavior if you set a local zone.
• Non-idempotent jobs. The DST gap/overlap is only visible as damage because the job double-charges or double-sends instead of deduping.
• Host clock or tzdata drift. An out-of-date tzdata package applies last year’s DST rules, shifting the transition by weeks.

Diagnostic Workflow

Confirm what timezone and DST rules the host is actually using:

timedatectl                       # Time zone line + "DST active" hint
date +'%Z %z %::z'                # current abbreviation and offset, e.g. EDT -0400
cat /etc/timezone 2>/dev/null

Check whether tzdata knows the correct transition dates (stale tzdata is a real cause):

zdump -v America/New_York | grep 2026    # prints the 2026 spring/fall transition instants
dpkg -l tzdata 2>/dev/null || rpm -q tzdata

Inspect how the schedule is declared. For Vixie/cron, look for a timezone override:

grep -rE 'CRON_TZ|^TZ=' /etc/crontab /etc/cron.d/ /var/spool/cron/ 2>/dev/null

For systemd timers, ask systemd to show the next elapse and the effective zone:

systemctl list-timers --all
systemctl cat billing-run.timer          # check OnCalendar= and Persistent=
systemd-analyze calendar --iterations=5 'OnCalendar=*-*-* 02:30:00 America/New_York'

For a Kubernetes CronJob, check whether the API server honors a timezone field:

kubectl get cronjob billing-run -o jsonpath='{.spec.timeZone}{"\n"}'
kubectl get cronjob billing-run -o jsonpath='{.status.lastScheduleTime}{"\n"}'

Cross-reference the job’s own execution log against the transition instant from zdump to prove a run landed in the gap (spring) or the repeated hour (fall).

Example Root Cause Analysis

A billing job scheduled as 30 2 * * * in America/New_York fed a “1 run per day” invariant. On 2026-03-08 (spring forward), local time jumped 02:00 to 03:00, so 02:30 never occurred and cron never fired the job — customers were under-billed for that day. Eight months later, on 2026-11-01 (fall back), local 01:00-02:00 occurred twice; an hourly-sync job that used 0 * * * * ran the 01:00 slot in both EDT and EST, double-inserting rows because it keyed on the wall-clock hour label, not an absolute instant.

zdump -v America/New_York confirmed the transition instants matched the missing and duplicated log entries exactly. The root cause was two-fold: schedules were expressed in a DST-observing zone, and the jobs treated the local wall-clock hour as a unique key. The fix was to move both schedules to UTC (CRON_TZ=UTC and a UTC OnCalendar), which has no gaps or overlaps, and to add an idempotency key based on the absolute UTC date so any accidental double-fire became a safe no-op.

Prevention Best Practices

• Schedule in UTC. UTC never changes offset, so it has no missing or repeated times. Set CRON_TZ=UTC, use UTC OnCalendar= values, and leave Kubernetes CronJobs on the default UTC clock. Convert the intended local time to UTC once, at design time.
• Make jobs idempotent. Key each run on an absolute instant or a business date (a UTC calendar day), so a duplicate fire dedupes and a caught-up run cannot double-apply.
• Avoid scheduling in the 01:00-03:00 local window for any job that must stay in a local zone — that band contains both the gap and the overlap.
• Keep tzdata current. Patch tzdata on every host and rebuild container images regularly; governments change DST rules with little notice.
• Be deliberate with Persistent=true. systemd Persistent timers catch up missed runs after downtime, which can look like an extra DST run; pair it with idempotency.
• Alert on run count, not just failures. A monitor that expects exactly one success per UTC day catches both the skipped and the doubled run — neither of which produces a non-zero exit code.

Quick Command Reference

# What zone/offset is the host on right now?
timedatectl; date +'%Z %z'

# Show the actual DST transition instants for a zone
zdump -v America/New_York | grep 2026

# Find local-timezone overrides in cron
grep -rE 'CRON_TZ|^TZ=' /etc/crontab /etc/cron.d/ /var/spool/cron/

# Inspect a systemd timer's zone and next run
systemctl cat billing-run.timer
systemd-analyze calendar 'OnCalendar=*-*-* 02:30:00 UTC'

# Kubernetes CronJob timezone + last schedule
kubectl get cronjob billing-run -o jsonpath='{.spec.timeZone} {.status.lastScheduleTime}{"\n"}'

# Verify tzdata is installed and current
zdump -v UTC | head -1; dpkg -l tzdata 2>/dev/null || rpm -q tzdata

Conclusion

DST-related skipped and duplicated cron runs are not scheduler bugs — they are the predictable consequence of expressing a schedule in a timezone that has a missing hour every spring and a repeated hour every fall. The durable fix is to schedule in UTC, which has neither, and to make every job idempotent on an absolute instant so an accidental double-fire is harmless. Add a “exactly one run per UTC day” monitor and these silent twice-a-year anomalies stop reaching production.