🚨 Automation Error: Kubernetes CronJob 'too many missed start times'

Overview

A Kubernetes CronJob that was firing reliably suddenly stops creating Jobs, and the CronJob controller records this event against the object:

Warning  FailedNeedsStart  cronjob-controller  Cannot determine if job needs to be started:
too many missed start times (> 100). Set or decrease .spec.startingDeadlineSeconds or check
clock skew

No Job is created, no Pod runs, and unless something is watching the schedule, the automation simply goes quiet. This is one of the most common “the cron just stopped and nobody noticed” failure modes in Kubernetes-based automation.

Symptoms

• A CronJob that previously ran on schedule stops creating Jobs entirely.
• kubectl get cronjob shows a LAST SCHEDULE timestamp that is hours or days old.
• The FailedNeedsStart warning above appears in kubectl describe cronjob.
• Downstream artifacts the job produces (reports, backups, syncs) stop appearing, often discovered only when someone asks “where’s last night’s backup?”
• After a control-plane or node outage, several CronJobs go silent at once.

Common Root Causes

• The controller was unable to run for a long window. If the CronJob controller (or the whole control plane) was down, paused, or overloaded past ~100 scheduled ticks, it can no longer decide which missed runs to start and gives up rather than firing a storm of backfills.
• A tight or missing startingDeadlineSeconds combined with a long outage. Without a deadline, the controller counts every missed start since the last successful run; once that count exceeds 100 it refuses to schedule.
• The CronJob was suspended and then resumed (spec.suspend: true) after a long pause, so a huge backlog of missed times accrues at resume.
• Clock skew between the API server and nodes, which makes the controller compute a nonsensical number of missed intervals.
• An over-aggressive schedule (for example * * * * *) where jobs pile up faster than they complete, so missed starts accumulate.
• The Job template is invalid, so starts fail repeatedly and the “missed” count climbs.

Diagnostic Workflow

Start by confirming the schedule state and reading the controller’s own events:

kubectl get cronjob -A
kubectl get cronjob my-job -o wide
kubectl describe cronjob my-job | sed -n '/Events/,$p'

Check whether the CronJob is suspended and what its deadline is set to:

kubectl get cronjob my-job \
  -o jsonpath='{.spec.suspend}{"  deadline="}{.spec.startingDeadlineSeconds}{"\n"}'

Compare the last scheduled time against now to see the size of the gap:

kubectl get cronjob my-job \
  -o jsonpath='{.status.lastScheduleTime}{"\n"}'
date -u +%Y-%m-%dT%H:%M:%SZ

Confirm the CronJob controller has been healthy — a long restart or outage is the usual trigger:

kubectl -n kube-system get pods | grep -i 'controller-manager\|scheduler'
kubectl -n kube-system logs -l component=kube-controller-manager --since=6h \
  | grep -i 'cronjob\|missed start'

Rule out clock skew, which the error message explicitly calls out:

# On each node
timedatectl status | grep -i 'synchronized\|NTP'
chronyc tracking 2>/dev/null | grep -i 'System time'

Check whether the Job template itself is valid by forcing one manual run:

kubectl create job --from=cronjob/my-job my-job-manual-001
kubectl get job my-job-manual-001 -o wide
kubectl describe job my-job-manual-001 | sed -n '/Events/,$p'

Example Root Cause Analysis

A team ran a backup CronJob on 0 * * * * (hourly). A cluster upgrade drained and rolled the control-plane nodes, and the kube-controller-manager was unavailable for most of a weekend. When it came back, kubectl describe cronjob backup showed:

Warning  FailedNeedsStart  cronjob-controller  Cannot determine if job needs to be started:
too many missed start times (> 100)

The CronJob had no startingDeadlineSeconds, so the controller counted every missed hourly tick since the last successful run. With the controller down far longer than 100 hours’ worth of accumulated ticks (compounded by an earlier suspend during the upgrade), the count blew past 100 and the controller refused to schedule any run — including the current one.

The fix had two parts. First, unblock the present: set a bounded startingDeadlineSeconds so the controller only considers recent missed starts, not the entire backlog:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup
spec:
  schedule: "0 * * * *"
  startingDeadlineSeconds: 200   # only start if within 200s of the scheduled time
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: registry.internal/backup:1.9.0

kubectl apply -f backup-cronjob.yaml

With a 200-second deadline the controller ignores every stale missed start, the count drops below 100, and the next top-of-hour tick schedules normally. Second, they added an alert on time() - kube_cronjob_status_last_schedule_time so a silent scheduler is caught within one interval instead of on Monday.

Prevention Best Practices

• Always set startingDeadlineSeconds to a value shorter than the schedule interval (but longer than normal scheduling latency). This bounds how far back the controller looks and prevents the >100 lockup after any outage.
• Choose concurrencyPolicy deliberately — Forbid to skip a run while the previous one is still going, or Replace to supersede it — so overruns don’t accumulate as missed starts.
• Alert on scheduler silence, not just on job failure. Track kube_cronjob_status_last_schedule_time (via kube-state-metrics) and page if now - last_schedule exceeds one interval plus a margin.
• Keep NTP healthy on all nodes and the control plane; the controller’s missed-start math depends on synchronized clocks.
• Avoid sub-minute or minute-level schedules for jobs that can take longer than the interval; they generate missed starts by design.
• Un-suspend carefully — after a long suspend, expect a backlog and rely on startingDeadlineSeconds to absorb it rather than backfilling.

Quick Command Reference

# Read the controller's decision and events
kubectl describe cronjob my-job | sed -n '/Events/,$p'

# See suspend flag and deadline
kubectl get cronjob my-job -o jsonpath='{.spec.suspend}{"  "}{.spec.startingDeadlineSeconds}{"\n"}'

# Compare last schedule to now
kubectl get cronjob my-job -o jsonpath='{.status.lastScheduleTime}{"\n"}'; date -u +%Y-%m-%dT%H:%M:%SZ

# Set a bounded deadline to break the >100 lockup
kubectl patch cronjob my-job --type merge -p '{"spec":{"startingDeadlineSeconds":200}}'

# Force one run to validate the Job template
kubectl create job --from=cronjob/my-job my-job-manual-001

# Check controller health and clock sync
kubectl -n kube-system logs -l component=kube-controller-manager --since=6h | grep -i cronjob
timedatectl status | grep -i synchronized

Conclusion

too many missed start times is the CronJob controller protecting you from a backfill storm after it couldn’t schedule for a long window — but the side effect is a silently dead schedule. The durable fix is to set a bounded startingDeadlineSeconds so the controller only ever considers recent ticks, pick a concurrencyPolicy that stops overruns from piling up, and — most importantly — alert on scheduler silence so a stopped cron is caught within one interval instead of days later. A CronJob that fails loudly is an inconvenience; one that stops quietly is an incident waiting to be discovered by its absence.