🚨 Kubernetes Error Guide: 'persistentvolumeclaim not found' StatefulSet Pod

Exact Error Message

A StatefulSet pod will not start. It stays Pending, and its events show the kubelet or scheduler cannot find the PersistentVolumeClaim the pod expects from its volumeClaimTemplates:

$ kubectl get pod kafka-2
NAME      READY   STATUS    RESTARTS   AGE
kafka-2   0/1     Pending   0          3m

$ kubectl describe pod kafka-2
Events:
  Type     Reason            Age                From                Message
  ----     ------            ----               ----                ----
  Warning  FailedScheduling  2m (x4 over 3m)    default-scheduler   0/3 nodes are available: persistentvolumeclaim "data-kafka-2" not found. preemption: 0/3 nodes are available...

The signal is persistentvolumeclaim "data-kafka-2" not found. The name follows the StatefulSet convention <volumeClaimTemplate-name>-<statefulset-name>-<ordinal> — here template data, StatefulSet kafka, ordinal 2.

What the Error Means

A StatefulSet’s volumeClaimTemplates create one PVC per pod, named deterministically by ordinal: data-kafka-0, data-kafka-1, data-kafka-2. The StatefulSet controller creates these PVCs as pods scale up, and expects them to exist when a pod is (re)scheduled. The scheduler refuses to place a pod whose declared PVC does not exist, producing persistentvolumeclaim "..." not found.

The controller only creates a missing template PVC during scale-up. If a PVC for an existing ordinal is deleted out from under the StatefulSet — by a cleanup job, a helm uninstall, or a manual kubectl delete pvc — the controller does not always recreate it cleanly, and the pod is stranded waiting for a claim that no longer exists. The deterministic name is both the diagnosis and the recovery key: recreate the exact PVC name and the pod binds.

Common Causes

• PVC manually deleted — someone ran kubectl delete pvc data-<sts>-<n> while the StatefulSet still references that ordinal.
• Helm/Kustomize cleanup removed PVCs — an uninstall or prune deleted the claims but left or recreated the StatefulSet.
• PVC reclaim/retain mismatch — the PV was Released and Retained, but the old PVC object is gone, so the pod’s expected claim name does not resolve.
• Scaled down then up with PVC pruning — a controller or policy deleted PVCs on scale-down (e.g. persistentVolumeClaimRetentionPolicy: Delete), and they are now expected again.
• Namespace mismatch / wrong template name — the volumeClaimTemplates metadata name does not match the volume mount, so the generated PVC name differs from what the pod references.
• Restored StatefulSet without its PVCs — the StatefulSet manifest was reapplied but the PVCs were never restored alongside it.

How to Reproduce the Error

Create a StatefulSet with a volume template, then delete one ordinal’s PVC and its pod:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka
  replicas: 3
  selector:
    matchLabels: { app: kafka }
  template:
    metadata:
      labels: { app: kafka }
    spec:
      containers:
        - name: broker
          image: registry.k8s.io/pause:3.9
          volumeMounts:
            - name: data
              mountPath: /var/lib/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

kubectl apply -f kafka-sts.yaml
kubectl delete pvc data-kafka-2
kubectl delete pod kafka-2
kubectl get pod kafka-2

NAME      READY   STATUS    RESTARTS   AGE
kafka-2   0/1     Pending   0          30s

The recreated kafka-2 cannot schedule: its PVC data-kafka-2 is gone.

Diagnostic Commands

# Read the not-found PVC name from the pod's events
kubectl describe pod <STS>-<ORDINAL> | grep -A6 Events

# Which PVCs actually exist for this StatefulSet?
kubectl get pvc -l app=<APP>

# Confirm the volume template name and expected claim naming
kubectl get statefulset <STS> -o jsonpath='{.spec.volumeClaimTemplates[*].metadata.name}'

# Look for an orphaned Released PV that held the old data
kubectl get pv | grep <NS>/<EXPECTED-PVC>

# Check the retention policy that may have pruned PVCs
kubectl get statefulset <STS> -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}'

Match the not-found name from the pod events against the actual PVC list to confirm exactly which claim is missing.

Step-by-Step Resolution

1. Read the exact missing PVC name. The event gives you the deterministic name, e.g. data-kafka-2. Confirm it is absent:

kubectl get pvc data-kafka-2
# Error from server (NotFound): persistentvolumeclaims "data-kafka-2" not found

2. Check for a retained PV holding the old data. If the original PV used Retain, it still exists in Released state with the real data. You can reattach it by recreating a PVC bound to that PV:

kubectl get pv | grep Released

3a. Reattach a retained PV. Clear the stale claimRef on the PV, then create a PVC with the exact expected name and matching storageClassName/size so it binds to that PV — preserving the data. Then delete the stuck pod so it reschedules.

3b. Recreate an empty PVC. If no PV survives (data is gone), create a fresh PVC with the exact name data-<sts>-<n> and the template’s spec. The pod binds and starts empty; restore data from backup.

4. Delete the stranded pod to retrigger scheduling. Once the PVC exists, recreate the pod so the scheduler re-evaluates:

kubectl delete pod kafka-2

5. Fix the root cause. If a cleanup job or persistentVolumeClaimRetentionPolicy: Delete removed the PVC, change the policy to Retain (the default behavior is to keep PVCs) and exclude StatefulSet PVCs from prune automation.

6. Verify binding and readiness. Confirm the PVC is Bound and the pod proceeds. The ordered rollout resumes from that ordinal.

Prevention and Best Practices

• Use Retain reclaim policy for stateful data so deleting a PVC never silently destroys the underlying PV.
• Keep persistentVolumeClaimRetentionPolicy at the conservative default; only set whenDeleted/whenScaled: Delete if you truly want storage reaped.
• Exclude StatefulSet PVCs from Helm/Kustomize prune and from any namespace cleanup tooling — they outlive pods by design.
• Back up the PVC objects (and the data) so a missing claim can be recreated with the exact name and bound to surviving PVs.
• Never kubectl delete pvc on a live StatefulSet ordinal; scale down first if you intend to release storage. More in Kubernetes & Helm guides.

Related Errors

• PersistentVolumeClaim not bound — the PVC exists but cannot bind to a PV.
• StatefulSet has not progressed — the rollout-stall this can trigger.
• FailedScheduling — the broader scheduling failure this message is a variant of.

Frequently Asked Questions

Why doesn’t the StatefulSet just recreate the missing PVC? The controller only auto-creates volumeClaimTemplates PVCs during scale-up for new ordinals. For an existing ordinal whose PVC was deleted, it expects the claim to already exist and will not reliably regenerate it, so the pod stalls.

How is the PVC name derived? It is <template-name>-<statefulset-name>-<ordinal>. A template named data in StatefulSet kafka produces data-kafka-0, data-kafka-1, and so on. The ordinal in the missing pod’s name tells you the exact PVC to recreate.

Can I recover the original data? Only if the underlying PV used the Retain policy and still exists in Released state. Recreate the PVC bound to that PV by clearing its claimRef. If the PV was deleted, the data is gone and you must restore from backup.

Is this the same as PVC not bound? No. not bound means the PVC object exists but has no matching PV. not found means the PVC object itself does not exist. Different fixes: provision/bind a PV versus recreate the claim.