Kubernetes Job Pod Failure Policy & Success Policy Design Prompt

You are a senior Kubernetes batch-workloads engineer. You run large Indexed Jobs (ETL, ML training, simulation) and know the exact semantics of `backoffLimit`, `podFailurePolicy`, `successPolicy`, `backoffLimitPerIndex`, and `maxFailedIndexes`. You know why a naive `backoffLimit: 6` wastes hours retrying a pod that hit a permanent OOM or a preempted spot node, and how to make Jobs distinguish "the code is wrong" from "the node died."

I will provide:
- Cluster version (podFailurePolicy/successPolicy GA depends on it)
- The Job spec (Job or Indexed Job, completions/parallelism)
- The failure modes I see (OOMKills, spot/node preemption, application exit codes, disruption evictions)
- What "success" means for this Job (all indexes, or a quorum/subset)

Your job:

1. **Diagnose the current retry behavior**:
   - Explain how `backoffLimit` counts *all* pod failures across indexes and why that conflates unrelated causes.
   - Identify which of their failures are retriable (node preemption, transient network) vs. non-retriable (bad input, permanent OOM, exit code that means "config wrong").

2. **Design a `podFailurePolicy`** with explicit rules and rationale for each:
   - `action: FailJob` on application exit codes that can never succeed on retry (fail fast, stop wasting compute).
   - `action: Ignore` on `DisruptionTarget` (graceful node shutdown, preemption) so infra churn does not consume the retry budget.
   - `action: Count` (default) for genuinely retriable app errors.
   - `action: FailIndex` (with `backoffLimitPerIndex`) so one poisoned shard does not sink the whole Job.
   - Match on `onExitCodes` and/or `onPodConditions` correctly; explain the operator (`In`/`NotIn`) and value semantics.

3. **Design `backoffLimitPerIndex` + `maxFailedIndexes`** for Indexed Jobs:
   - Per-index retry budget so a bad shard fails in isolation.
   - `maxFailedIndexes` to abort early once too many shards are unrecoverable.

4. **Design a `successPolicy`** where partial completion is acceptable:
   - `succeededIndexes` and/or `succeededCount` so the Job is marked complete when a quorum finishes, and remaining pods are terminated (with the `SuccessCriteriaMet` condition).
   - Call out the compute savings and when this is unsafe (e.g., every shard's output is required).

5. **Produce the full corrected Job manifest** and a short table mapping each failure mode → chosen action → why.

6. **Validation & observability**:
   - Commands to confirm the policy fired (`kubectl get job -o yaml` conditions, pod `status.conditions`, events).
   - Metrics/alerts to watch (`FailJob` firing, `maxFailedIndexes` hit).

Mark DESTRUCTIVE / footgun: `action: FailJob` on an exit code that is actually transient (kills otherwise-recoverable runs), `action: Ignore` used too broadly (masks real failures and can loop indefinitely on a bad node), and `successPolicy` on a Job where every index's output is genuinely required (silently discards work).

---

Cluster version: [DESCRIBE]
Job spec:
```yaml
[PASTE Job / Indexed Job spec]
```
Failure modes observed: [DESCRIBE — exit codes, OOM, preemption, evictions]
Definition of success (all indexes / quorum / subset): [DESCRIBE]

Why this prompt works

The default backoffLimit treats every pod failure the same, so a spot-node preemption and a “your config is broken” exit code both eat the same budget. podFailurePolicy and successPolicy let a Job react to why a pod failed. This prompt forces the assistant to classify each failure mode and pick the right action, which is the whole point — and where most teams get it wrong.

How to use it

1. List real exit codes and reasons you have seen, not hypotheticals. The policy is only as good as the mapping.
2. Say plainly whether partial completion is acceptable — that decides whether successPolicy is on the table.
3. Confirm your pods use restartPolicy: Never so onExitCodes matching behaves predictably.

Useful commands

# Inspect the Job's policy and current conditions
kubectl get job <name> -o yaml | yq '.spec.podFailurePolicy, .spec.successPolicy, .status.conditions'

# Per-index state on an Indexed Job
kubectl get job <name> -o jsonpath='{.status.completedIndexes} {.status.failedIndexes}{"\n"}'

# Why a specific pod failed (condition + exit code)
kubectl get pod <pod> -o jsonpath='{.status.conditions}'
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

# Events for FailJob / disruption
kubectl get events --field-selector involvedObject.name=<pod> --sort-by='.lastTimestamp'

Pattern: Indexed Job with failure + success policies

apiVersion: batch/v1
kind: Job
metadata:
  name: ml-shard-train
spec:
  completions: 50
  parallelism: 10
  completionMode: Indexed
  backoffLimit: 20                 # global cap; per-index budget below is the real control
  backoffLimitPerIndex: 3          # each shard retries up to 3 times on its own
  maxFailedIndexes: 5              # abort the Job once 5 shards are unrecoverable
  podFailurePolicy:
    rules:
    - action: FailJob              # exit 42 = bad config, never succeeds on retry
      onExitCodes:
        operator: In
        values: [42]
    - action: Ignore               # node preemption / graceful shutdown: don't spend budget
      onPodConditions:
      - type: DisruptionTarget
    - action: FailIndex            # exit 137 (OOM): fail just this shard's index
      onExitCodes:
        operator: In
        values: [137]
  successPolicy:
    rules:
    - succeededCount: 45           # 45/50 shards done = good enough; stop the rest
  template:
    spec:
      restartPolicy: Never         # required for predictable onExitCodes matching
      containers:
      - name: train
        image: myorg/trainer:v3
        command: ["/usr/local/bin/train.sh"]

Common findings this catches

• Spot preemptions eating backoffLimit → add action: Ignore on DisruptionTarget.
• Bad input looping for hours → add action: FailJob on the deterministic exit code.
• One poisoned shard failing the whole Job → FailIndex + backoffLimitPerIndex.
• restartPolicy: OnFailure → switch to Never for reliable exit-code matching.
• All-or-nothing Job that could accept a quorum → successPolicy.succeededCount saves compute.

When to escalate

• Feature not enabled on the control plane — coordinate a cluster upgrade or gate change.
• Repeated OOM FailIndex on the same shards — the workload needs right-sizing or data fixes, not more retries.
• maxFailedIndexes firing regularly — investigate node health or input quality before loosening the limit.