🚨 Kubernetes CronJob Error: DeadlineExceeded Kills Jobs Before Completion

Overview

A Kubernetes Job (including one created by a CronJob) that runs longer than its activeDeadlineSeconds is forcibly terminated and marked failed with DeadlineExceeded. The Job’s status shows:

Status:         Failed
Reason:         DeadlineExceeded
Message:        Job was active longer than specified deadline

And the pod events show the termination:

Warning  DeadlineExceeded  job-controller  Job was active longer than specified deadline

The work is cut off mid-execution — a backup that never finishes, a batch that processes half its records — even though nothing in the job itself errored.

Symptoms

• A CronJob’s Job ends in Failed with Reason: DeadlineExceeded.
• The pod is terminated while still running, not on a non-zero exit.
• Failures started after the workload grew (more data, slower dependency) but the deadline stayed fixed.
• Partial side effects: half-written output, incomplete batch, an interrupted migration.
• A separate, distinct error — Cannot determine if job needs to be started ... too many missed start times — points instead at startingDeadlineSeconds, not activeDeadlineSeconds.

Common Root Causes

• activeDeadlineSeconds too short — the deadline was set for the original runtime and the job now legitimately takes longer.
• Slow or degraded dependency — the job waits on a database, API, or volume that got slower, pushing runtime past the deadline.
• Data growth — the input set grew and the job’s per-run duration scaled with it.
• Confusing the two deadlines — startingDeadlineSeconds (how late a run may start) is mistaken for a runtime limit, or vice versa.
• No progress/parallelism — a serial job that could be sharded runs long enough to hit the wall.
• Deadline shorter than the schedule interval expectation — the job simply cannot finish its work in the allotted time.

Diagnostic Workflow

Confirm the failure reason on the Job:

kubectl get job <job> -o jsonpath='{.status.conditions}' | jq
kubectl describe job <job> | grep -iA2 'DeadlineExceeded\|Active Deadline'

Read the two deadline settings on the CronJob and its Job template:

kubectl get cronjob <name> -o jsonpath='{.spec.jobTemplate.spec.activeDeadlineSeconds}'; echo
kubectl get cronjob <name> -o jsonpath='{.spec.startingDeadlineSeconds}'; echo

Measure how long the job actually needs versus the deadline:

kubectl get job <job> -o jsonpath='{.status.startTime}'; echo
# compare completion/termination time from events
kubectl describe job <job> | grep -iE 'Started|Duration|Deadline'

Look at the pod logs to see where the time went (slow step vs. steady progress):

kubectl logs job/<job> --tail=200 --timestamps

Check whether a slow dependency, not the job, is the cause:

kubectl logs job/<job> | grep -iE 'timeout|retry|waiting for|slow'

Example Root Cause Analysis

A nightly db-export CronJob started failing with DeadlineExceeded. kubectl describe job confirmed the reason, and the CronJob had activeDeadlineSeconds: 600 set months earlier when the export took ~4 minutes. Pod logs with timestamps showed the export now ran 11 minutes before being killed at the 10-minute mark:

2026-07-09T02:00:03Z exporting table events (18.2M rows)
2026-07-09T02:09:58Z ... 82% complete
<terminated: DeadlineExceeded>

Nothing was broken — the dataset had simply grown, and the fixed 600-second deadline no longer fit. Because the export left a partial file, each failed run also wasted the next run’s time cleaning up.

The fix raised activeDeadlineSeconds to give real headroom over the observed runtime and added startingDeadlineSeconds so a briefly delayed start would not be skipped. For durability, the export was made resumable (write to a temp object, atomically promote on success) so a future overrun cannot leave a half-written file, and a follow-up sharded the export by table so per-Job runtime stays bounded as data grows.

Prevention Best Practices

• Size activeDeadlineSeconds to the observed runtime plus generous headroom, and revisit it as data grows — do not leave the original value in place forever.
• Do not confuse the two deadlines: activeDeadlineSeconds limits how long a run may execute; startingDeadlineSeconds limits how late it may start.
• Make jobs resumable or idempotent so a termination never leaves a partial, unrecoverable result.
• Shard long jobs (by table, key range, or time bucket) so per-Job runtime stays bounded.
• Alert on runtime trending toward the deadline so you raise it before jobs start dying.
• Investigate slow dependencies rather than only extending the deadline when a job suddenly runs long.

Quick Command Reference

# Confirm DeadlineExceeded
kubectl describe job <job> | grep -iA2 'DeadlineExceeded'

# Read both deadlines
kubectl get cronjob <name> -o jsonpath='{.spec.jobTemplate.spec.activeDeadlineSeconds}'; echo
kubectl get cronjob <name> -o jsonpath='{.spec.startingDeadlineSeconds}'; echo

# See where the runtime went
kubectl logs job/<job> --tail=200 --timestamps

# Patch a longer active deadline
kubectl patch cronjob <name> --type merge \
  -p '{"spec":{"jobTemplate":{"spec":{"activeDeadlineSeconds":1800}}}}'

Conclusion

DeadlineExceeded means the Job did nothing wrong except take longer than you allowed. It is usually a stale activeDeadlineSeconds colliding with a workload that grew or a dependency that slowed. Raise the deadline to fit reality with headroom, keep it distinct from startingDeadlineSeconds, and make the job resumable and shardable so growth does not put it back on a collision course with the wall — and so a rare overrun never leaves half-finished work behind.