Kubernetes Job & CronJob Debug Prompt

You are a senior Kubernetes engineer who has run thousands of batch Jobs and scheduled CronJobs in production. You know the controller's quirks — `concurrencyPolicy`, missed runs after pause, timezone defaults, completions vs parallelism — and how to debug a Job that won't terminate.

I will provide:
- The Job or CronJob name + namespace
- The symptom (Job stuck in `Active`, pods retrying forever, CronJob missed schedule, history not cleaning up, runs overlap)
- `kubectl get job <name> -o yaml` or `kubectl get cronjob <name> -o yaml`
- Pod status from `kubectl get pods -l job-name=<name>`
- For CronJob: the schedule, `concurrencyPolicy`, `startingDeadlineSeconds`, time zone config

Your job:

1. **For Jobs**:
   - **`completions`** — how many pods must complete successfully (default 1)
   - **`parallelism`** — how many pods run at once (default 1)
   - **`backoffLimit`** — total pod failures before Job declared failed (default 6)
   - **`activeDeadlineSeconds`** — max wall-clock time; cuts off the Job at deadline
   - **`ttlSecondsAfterFinished`** — auto-cleanup after completion (set; default unlimited persistence)
   - **`completionMode`** — `NonIndexed` (default) or `Indexed` (each pod gets a unique index for sharding)
2. **Job state transitions**:
   - `Active` → pods running
   - `Succeeded` → enough pods completed (`completions` reached)
   - `Failed` → backoffLimit exceeded OR activeDeadline hit
   - The Job object stays around (consuming etcd / cluttering kubectl) unless TTL is set
3. **For "Job stuck Active forever"**:
   - Pods may be failing and being retried up to `backoffLimit`
   - Check pod logs from FAILED attempts (`kubectl logs <pod> --previous`)
   - Active deadline not set → no timeout
   - Pod can't terminate (stuck in finalizer, volume detach) → Job sees it as still active
4. **For CronJobs**:
   - **`schedule`** — cron format (5 fields: minute hour dayOfMonth month dayOfWeek)
   - **`timeZone`** (1.25+) — default UTC. Use explicit time zone like `America/New_York`.
   - **`concurrencyPolicy`** — `Allow` (default, runs overlap), `Forbid` (skip if previous still running), `Replace` (kill previous and start new)
   - **`startingDeadlineSeconds`** — how late can the CronJob controller still start a missed run (default unlimited; recommend setting)
   - **`suspend`** — pause the CronJob without deleting
   - **`successfulJobsHistoryLimit`** / **`failedJobsHistoryLimit`** — how many old Jobs to keep (default 3 success, 1 fail)
5. **For CronJob missed schedules**:
   - The CronJob controller checks every 10 seconds; small drift
   - If too many missed runs (>100) in the deadline window, the controller stops trying — alerts and check logs
   - Time zone mismatch — schedule `0 9 * * *` is 09:00 UTC by default; users expecting local time get surprised
6. **For overlap issues**:
   - `concurrencyPolicy: Allow` lets runs overlap → cumulative load if jobs run long
   - `concurrencyPolicy: Forbid` prevents overlap but causes drift if a single run is long
   - `concurrencyPolicy: Replace` kills the previous run — only safe if your job is idempotent
7. **For "old Jobs piling up"**:
   - Set `ttlSecondsAfterFinished` on Job spec
   - For CronJobs, lower `successfulJobsHistoryLimit` / `failedJobsHistoryLimit`
   - Manually clean: `kubectl delete jobs --field-selector status.successful=1 -n <ns>`

Mark DESTRUCTIVE: setting `ttlSecondsAfterFinished` very low while debugging (pod logs and Job objects auto-delete; lose forensics), `concurrencyPolicy: Replace` on non-idempotent jobs.

---

Workload: [Job / CronJob]
Name + namespace: [DESCRIBE]
Symptom: [DESCRIBE]
`kubectl get job <name> -o yaml` or `cronjob`:
```yaml
[PASTE relevant spec + status]
```
Pods: `kubectl get pods -l job-name=<name>`:
```
[PASTE]
```
For CronJob: schedule, timezone, concurrencyPolicy:
[DESCRIBE]

Why this prompt works

Jobs and CronJobs are simpler than other workloads but have subtle quirks: time zones, missed-run behavior, completion semantics, history cleanup. Misunderstanding any of these results in silent failures or runaway resource consumption.

How to use it

1. Read completions and parallelism together — they define the success criteria.
2. For CronJobs, always state the time zone you expect. The default is UTC.
3. Check pod logs from PREVIOUS attempts when backoff is happening (kubectl logs --previous).
4. For cleanup issues, look at ttlSecondsAfterFinished and historyLimit.

Useful commands

# Jobs
kubectl get jobs -A
kubectl describe job <name>
kubectl get job <name> -o yaml | yq '.spec, .status'

# Pods of a Job
kubectl get pods -l job-name=<name>
kubectl logs <pod>
kubectl logs <pod> --previous          # logs from a CrashLoop attempt

# CronJobs
kubectl get cronjobs -A
kubectl describe cronjob <name>
kubectl get cronjob <name> -o yaml

# Manually trigger a CronJob (creates a Job)
kubectl create job manual-$(date +%s) --from=cronjob/<name>

# Suspend / resume
kubectl patch cronjob <name> -p '{"spec":{"suspend":true}}'
kubectl patch cronjob <name> -p '{"spec":{"suspend":false}}'

# Cleanup
kubectl delete jobs --field-selector status.successful=1 -n <ns>
kubectl delete jobs --field-selector status.successful=0 -n <ns>    # failed only

# Force-delete stuck Job
kubectl delete job <name> --grace-period=0 --force

# Cron validation (validate the schedule string)
# (Use external tools like crontab.guru; K8s doesn't validate beyond syntax)

Pattern: ETL Job

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-etl
spec:
  parallelism: 1
  completions: 1
  backoffLimit: 3
  activeDeadlineSeconds: 7200       # 2 hours max
  ttlSecondsAfterFinished: 86400     # auto-clean after 1 day
  template:
    spec:
      restartPolicy: OnFailure       # Job-pod policy; Never or OnFailure
      containers:
      - name: etl
        image: myorg/etl:v1
        command: ["/usr/local/bin/run-etl.sh"]

Pattern: Daily CronJob with safe defaults

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-report
spec:
  schedule: "0 3 * * *"              # 03:00 (UTC default; or set timeZone)
  timeZone: "UTC"
  concurrencyPolicy: Forbid           # skip if previous still running
  startingDeadlineSeconds: 300        # if missed by >5 min, skip
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      activeDeadlineSeconds: 3600
      ttlSecondsAfterFinished: 86400
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: report
            image: myorg/report:v1
            command: ["/usr/local/bin/daily.sh"]

Pattern: Indexed Job (sharded batch)

apiVersion: batch/v1
kind: Job
metadata:
  name: shard-process
spec:
  completions: 10
  parallelism: 5
  completionMode: Indexed             # each pod gets JOB_COMPLETION_INDEX env
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: worker
        image: myorg/worker:v1
        command:
        - sh
        - -c
        - 'echo "Processing shard $JOB_COMPLETION_INDEX"; ./process.sh $JOB_COMPLETION_INDEX'

Common findings this catches

• CronJob fires at “wrong time” → schedule interpreted as UTC; add timeZone field (1.25+) or adjust schedule.
• concurrencyPolicy: Allow + slow job → multiple runs stack up; switch to Forbid or Replace.
• Job stuck Active → pods CrashLoopBackOff; check kubectl logs --previous; raise backoffLimit only if transient.
• History piling up → set ttlSecondsAfterFinished on Job spec or lower history limits on CronJob.
• Missed schedules after cluster outage → if startingDeadlineSeconds unset, controller may fire many runs at once; set a sane deadline.
• restartPolicy: Always in Job pod spec — invalid for Jobs (Job pods must be OnFailure or Never).
• ttlSecondsAfterFinished: 0 during debugging → Job and pods vanish before you can read logs.

When to escalate

• CronJob controller errors in kube-controller-manager — cluster admin.
• Suspected DOS from missed-run catch-up — temporarily suspend the CronJob, then plan a clean restart.
• Indexed Job semantics issues (sharding logic) — application code; coordinate with app team.