🚨 Kubernetes Error Guide: 'secret not found' — Restore the Reference

Overview

When a pod references a Secret that does not exist in its namespace, the kubelet cannot assemble the container’s configuration and refuses to start it. The pod schedules onto a node, but the container stays in Waiting with the reason CreateContainerConfigError, and the events name the missing object: secret "x" not found. A missing imagePullSecret is a related variant — there the node cannot authenticate to a private registry and the image pull fails instead. Either way, nothing crashes; the container is simply never created because a required input is absent.

You will see this in the pod status column:

NAME                        READY   STATUS                       RESTARTS   AGE
api-8b6d4f9c7a-t5r2m        0/1     CreateContainerConfigError   0          1m50s

Secrets are namespaced, so a reference resolves only within the pod’s own namespace. The usual causes are: the Secret was never created (or was deleted), it lives in a different namespace than the pod, a key named in a valueFrom.secretKeyRef does not exist, a name typo, or a type mismatch (for example an imagePullSecrets entry that points at a Secret that is not kubernetes.io/dockerconfigjson). Since the pod is already scheduled, the fix is always about making the referenced Secret exist, with the right type and key, where the pod looks for it.

Symptoms

• Pod status is CreateContainerConfigError with READY 0/1; the container never reaches Running.
• For a missing pull secret, status is ErrImagePull/ImagePullBackOff with an auth-related message.
• kubectl describe pod events show Error: secret "x" not found or couldn't find key <k> in Secret <ns>/<name>.
• The Secret is missing from kubectl get secret -n <ns>, or exists in a different namespace.

kubectl get pods -l app=api -n prod

NAME                        READY   STATUS                       RESTARTS   AGE
api-8b6d4f9c7a-t5r2m        0/1     CreateContainerConfigError   0          1m50s

kubectl describe pod api-8b6d4f9c7a-t5r2m -n prod | grep -A4 Events

Events:
  Type     Reason     Age               From      Message
  ----     ------     ----              ----      -------
  Warning  Failed     9s (x8 over 1m)   kubelet   Error: secret "api-db-creds" not found

Common Root Causes

1. The Secret was never created or was deleted

The pod references api-db-creds, but no such Secret exists in the namespace — it was forgotten in the manifest set, or was managed out-of-band (sealed-secrets, external-secrets) and never synced.

kubectl get secret -n prod

NAME                  TYPE                                  DATA   AGE
default-token-abc12   kubernetes.io/service-account-token   3      30d

api-db-creds is absent. Create the Secret (or fix the sync source), and the container is created on the next kubelet retry.

2. The Secret is in the wrong namespace

Secrets do not cross namespaces. A Secret in default cannot be referenced by a pod in prod — a common slip when promoting manifests between environments.

kubectl get secret api-db-creds --all-namespaces

NAMESPACE   NAME           TYPE     DATA   AGE
default     api-db-creds   Opaque   2      10d

The Secret exists, just not in prod. Recreate it in the pod’s namespace.

3. A referenced key does not exist in the Secret

A valueFrom.secretKeyRef names both a Secret and a key. If the Secret exists but the key does not (renamed, typo’d), the container still fails to be created.

kubectl describe pod api-8b6d4f9c7a-t5r2m -n prod | grep -A3 Events

  Warning  Failed   11s   kubelet   Error: couldn't find key PASSWORD in Secret prod/api-db-creds

Add the missing key to the Secret, or fix the key: in the pod’s secretKeyRef.

4. A missing or wrong-type imagePullSecret

If imagePullSecrets names a Secret that does not exist — or one that is not of type kubernetes.io/dockerconfigjson — the node cannot authenticate to the private registry and the image pull fails.

kubectl describe pod api-8b6d4f9c7a-t5r2m -n prod | grep -A3 Events

  Warning  Failed   14s   kubelet   Failed to pull image "registry.example.com/api:v3":
           failed to authorize: unauthorized

Confirm the pull secret exists and is a docker-registry Secret; create it with kubectl create secret docker-registry if not.

5. A name typo or wrong Secret type

The reference may be misspelled, or the Secret may exist with the wrong type for its use — a TLS mount expecting kubernetes.io/tls, or a pull secret that was created as Opaque.

kubectl get secret api-tls -n prod -o jsonpath='{.type}{"\n"}'

Opaque

A TLS volume expects kubernetes.io/tls (with tls.crt/tls.key), not Opaque. Recreate the Secret with the correct type.

Diagnostic Workflow

Step 1: Confirm the config error

kubectl get pod <POD> -n <NS> -o wide

CreateContainerConfigError points at a missing Secret or ConfigMap; ImagePullBackOff with an auth message points at a missing/invalid imagePullSecret.

Step 2: Read the exact message in events

kubectl describe pod <POD> -n <NS>

The Events section names the object: secret "x" not found, couldn't find key <k> in Secret <ns>/<name>, or an unauthorized pull failure.

Step 3: Check whether the Secret exists in this namespace

kubectl get secret -n <NS>
kubectl get secret <NAME> --all-namespaces

The first confirms presence in the pod’s namespace; the second reveals a copy in the wrong namespace.

Step 4: Inspect how the pod references the Secret

kubectl get pod <POD> -n <NS> -o jsonpath='{range .spec.containers[*]}{.env}{.envFrom}{"\n"}{end}'
kubectl get pod <POD> -n <NS> -o jsonpath='{.spec.volumes}{"\n"}'
kubectl get pod <POD> -n <NS> -o jsonpath='{.spec.imagePullSecrets}{"\n"}'

Look at every secretKeyRef, envFrom.secretRef, volumes[].secret, and imagePullSecrets entry for the exact name and key.

Step 5: Verify the Secret’s type and keys

kubectl get secret <NAME> -n <NS> -o jsonpath='{.type}{"\n"}'
kubectl get secret <NAME> -n <NS> -o jsonpath='{.data}{"\n"}' | tr ',' '\n'

Confirm the type matches the use (kubernetes.io/dockerconfigjson for pull secrets, kubernetes.io/tls for TLS) and that every expected key is present. data is base64 — you are checking key names, not values.

Example Root Cause Analysis

An api Deployment is rolled out and its pods will not start.

kubectl get pods -l app=api -n prod

NAME                        READY   STATUS                       RESTARTS   AGE
api-8b6d4f9c7a-t5r2m        0/1     CreateContainerConfigError   0          1m50s

The events name the missing object:

kubectl describe pod api-8b6d4f9c7a-t5r2m -n prod | grep -A2 Events

  Warning  Failed   7s   kubelet   Error: secret "api-db-creds" not found

The Secret is absent from prod, but a cluster-wide lookup finds it:

kubectl get secret api-db-creds --all-namespaces

NAMESPACE   NAME           TYPE     DATA   AGE
staging     api-db-creds   Opaque   2      8d

The Deployment was promoted from staging, but the Secret was managed separately and never created in prod. Because Secrets are namespaced, the prod pod has nothing to resolve, so the container is never created.

Fix: create the Secret in the pod’s namespace from the real source (here re-created explicitly rather than copied around), then re-roll:

kubectl create secret generic api-db-creds -n prod \
  --from-literal=USERNAME=api \
  --from-literal=PASSWORD='<from-vault>'
kubectl rollout restart deployment api -n prod

The container is created and the pod reaches 1/1 Running.

Prevention Best Practices

• Deploy Secrets and the workloads that reference them into the same namespace as one unit so a promotion never leaves credentials behind in another environment.
• Remember Secrets are namespaced and cannot be shared across namespaces — recreate them per environment rather than pointing across.
• Create pull secrets with kubectl create secret docker-registry so they get the correct kubernetes.io/dockerconfigjson type that imagePullSecrets requires.
• Match the Secret type to its use — kubernetes.io/tls for TLS volumes, dockerconfigjson for registries — a wrong type fails as surely as a missing Secret.
• Manage Secrets with a controller (External Secrets, Sealed Secrets) so the referenced object is reconciled into the namespace automatically instead of by hand.
• Keep secretKeyRef.key values in sync with the actual keys in the Secret, and validate manifests with the Kubernetes validator. See more in Kubernetes & Helm guides.

Quick Command Reference

# Confirm the config error (not a crash)
kubectl get pod <POD> -n <NS> -o wide

# Exact missing object / key in events
kubectl describe pod <POD> -n <NS>

# Does the Secret exist in this namespace? Anywhere?
kubectl get secret -n <NS>
kubectl get secret <NAME> --all-namespaces

# How the pod references it (env, volumes, pull secrets)
kubectl get pod <POD> -n <NS> -o jsonpath='{range .spec.containers[*]}{.env}{.envFrom}{"\n"}{end}'
kubectl get pod <POD> -n <NS> -o jsonpath='{.spec.volumes}{"\n"}'
kubectl get pod <POD> -n <NS> -o jsonpath='{.spec.imagePullSecrets}{"\n"}'

# Secret type and key names (data is base64)
kubectl get secret <NAME> -n <NS> -o jsonpath='{.type}{"\n"}'
kubectl get secret <NAME> -n <NS> -o jsonpath='{.data}{"\n"}' | tr ',' '\n'

# Create a registry pull secret
kubectl create secret docker-registry <NAME> -n <NS> \
  --docker-server=<REGISTRY> --docker-username=<USER> --docker-password=<PASS>

# Re-roll after a fix
kubectl rollout restart deployment <DEPLOY> -n <NS>

Conclusion

A secret "x" not found / CreateContainerConfigError pod is scheduled but cannot have its container created because a referenced Secret is missing. The usual root causes:

1. The Secret was never created or was deleted while a workload still references it.
2. The Secret exists but in a different namespace — Secrets do not cross namespaces.
3. The Secret exists but a referenced key (secretKeyRef.key) does not.
4. A missing or wrong-type imagePullSecret blocks the image pull with an auth error.
5. A name typo or a Secret created with the wrong type for its use.

Start with kubectl describe pod for the exact object name, then confirm with kubectl get secret -n <ns> (and --all-namespaces, plus a type check) — those steps identify almost every case. For quick triage, the free incident assistant can turn a config-error describe dump into the likely fix.