🚨 Kubernetes Error Guide: 'field is immutable' — Fix Forbidden Update Failures

Overview

Kubernetes rejects your apply or edit with a Forbidden validation error because you tried to change a field the API server treats as immutable. Certain fields are fixed for the life of an object — changing them would require rebuilding the object rather than patching it — so the API server refuses the update instead of silently doing something unexpected.

The literal error looks like this:

The Deployment "checkout-api" is invalid: spec.selector: Invalid value:
v1.LabelSelector{...}: field is immutable

You will also see it as spec.selector: Invalid value: ...: field is immutable for Services, spec.resources.requests.storage shrink attempts on PVCs, spec.template changes on Jobs, and spec.volumeName/spec.storageClassName on bound PVCs. In every case the message is the same shape: a specific field path, then field is immutable. The fix is never to “force” the patch — it is to recreate the object, or to stop changing that field.

Symptoms

• kubectl apply / kubectl edit / helm upgrade fails with Invalid value: ...: field is immutable.
• The rest of the manifest is valid; only one field path is called out.
• The live object is unchanged — the update was atomically rejected.
• Common offenders: Deployment/StatefulSet spec.selector, Service spec.clusterIP, Job spec.template, PVC spec.resources.requests.storage (shrink) and spec.storageClassName.

kubectl apply -f deployment.yaml

The Deployment "checkout-api" is invalid: spec.selector: Invalid value:
v1.LabelSelector{MatchLabels:map[string]string{"app":"checkout"}, ...}: field is immutable

Common Root Causes

1. Changing a Deployment or StatefulSet selector

spec.selector binds a controller to its pods and cannot change, because doing so would orphan the existing ReplicaSet/pods. Editing matchLabels (or the pod template labels the selector depends on) triggers the error.

kubectl get deployment checkout-api -o jsonpath='{.spec.selector.matchLabels}'

{"app":"checkout"}

If your new manifest sets app: checkout-api, the selector no longer matches and the update is forbidden. You must delete and recreate the Deployment (or keep the original selector).

2. Editing a Job’s pod template

A Job’s spec.template and most of its spec are immutable once created — a Job represents a fixed unit of work. Re-applying a Job with a changed image or command hits field is immutable.

The Job "db-migrate" is invalid: spec.template: Invalid value: ...: field is immutable

Delete the Job and create a new one (or use a generateName/versioned name so each run is a distinct object).

3. Shrinking a PVC or changing its StorageClass

PVCs support expansion but never shrinking, and spec.storageClassName/spec.volumeName are fixed once bound.

persistentvolumeclaims "data-0" is forbidden: spec.resources.requests.storage:
field is immutable ... only expansion is allowed

Set the request back to the current size (or larger), and never lower it.

4. Reassigning a Service’s clusterIP or type transitions

spec.clusterIP is immutable; you cannot change an assigned ClusterIP, and some type transitions are restricted.

Service "api" is invalid: spec.clusterIP: Invalid value: "10.96.10.5": field is immutable

Omit clusterIP from the manifest so the server keeps the existing one, or recreate the Service to get a new IP.

5. Helm upgrade carrying an immutable change

helm upgrade fails when the new chart version changes an immutable field (a common one: a selector label templated from a value you changed). The error surfaces the same field path.

Error: UPGRADE FAILED: cannot patch "checkout-api" with kind Deployment:
Deployment.apps "checkout-api" is invalid: spec.selector: ... field is immutable

Diagnostic Workflow

Step 1: Read the exact field path

kubectl apply -f manifest.yaml

The message names the precise field (spec.selector, spec.template, spec.resources.requests.storage). That path tells you which resource must be recreated versus reverted.

Step 2: Compare live vs desired for that field

kubectl get <kind> <name> -o jsonpath='{.spec.selector}'
kubectl diff -f manifest.yaml

kubectl diff shows exactly what your manifest would change; confirm the immutable field is the one that differs.

Step 3: Decide revert vs recreate

If the change was unintentional (e.g. a renamed label), revert that field to match live and re-apply. If the change is required, plan a recreate:

kubectl get deployment checkout-api -o yaml > /tmp/checkout-api.yaml
kubectl delete deployment checkout-api
kubectl apply -f manifest.yaml

Step 4: For Helm, inspect the rendered diff

helm diff upgrade checkout ./chart -n prod
helm template checkout ./chart | grep -A3 'selector'

Identify which value drives the immutable field and either keep it stable or plan a helm uninstall/helm install for that release.

Step 5: Verify the object after recreate

kubectl rollout status deployment checkout-api
kubectl get deployment checkout-api -o jsonpath='{.spec.selector.matchLabels}'

Example Root Cause Analysis

A team renames their app label from checkout to checkout-api across all manifests and runs the deploy. Every other resource applies, but the Deployment fails:

kubectl apply -f k8s/

The Deployment "checkout-api" is invalid: spec.selector: Invalid value:
v1.LabelSelector{MatchLabels:map[string]string{"app":"checkout-api"}}: field is immutable

Checking the live selector confirms the mismatch:

kubectl get deployment checkout-api -o jsonpath='{.spec.selector.matchLabels}'

{"app":"checkout"}

The selector was app: checkout when the Deployment was created; the new manifest changes it to app: checkout-api. Because the selector is immutable, the update is forbidden. The safe fix is a controlled recreate during a maintenance window (or a blue/green with a new Deployment name):

kubectl delete deployment checkout-api
kubectl apply -f k8s/deployment.yaml
kubectl rollout status deployment checkout-api

The new Deployment comes up with the renamed selector, and a fresh ReplicaSet is created cleanly.

Prevention Best Practices

• Treat selectors as permanent: pick label keys/values that will not need to change, since fixing them later requires a recreate.
• Use kubectl diff -f in CI before apply so an accidental immutable-field change is caught as a diff rather than a runtime failure.
• Version Job names (generateName or a -<timestamp> suffix) so re-runs create new objects instead of patching an immutable one.
• Omit server-assigned fields (clusterIP, spec.volumeName) from source manifests so the API server keeps the existing value.
• For Helm, avoid templating selector labels from values that change between releases. See more in Kubernetes & Helm guides.

Quick Command Reference

# See which field is rejected
kubectl apply -f manifest.yaml

# Compare live vs desired
kubectl diff -f manifest.yaml
kubectl get <kind> <name> -o jsonpath='{.spec.selector}'

# Back up before recreating
kubectl get deployment <name> -o yaml > /tmp/<name>.yaml

# Recreate when the change is required
kubectl delete deployment <name>
kubectl apply -f manifest.yaml
kubectl rollout status deployment <name>

# Helm: preview the immutable change
helm diff upgrade <release> ./chart -n <ns>

Conclusion

field is immutable means you tried to change a field Kubernetes fixes at creation time. The usual causes:

1. Editing a Deployment/StatefulSet spec.selector (or the labels it matches).
2. Re-applying a Job with a changed spec.template.
3. Shrinking a PVC or changing its storageClassName/volumeName.
4. Reassigning a Service clusterIP.
5. A helm upgrade that carries any of the above through a values change.

Read the exact field path, decide whether to revert or recreate, and use kubectl diff to catch these before they reach the cluster. For ad-hoc triage, the free incident assistant can turn a Forbidden apply error into the specific recreate plan.