What actually breaks, and what to check first
Kubernetes failures look alike and are not. Four completely different problems all present as "the pod is not running": Pending is the scheduler, ImagePullBackOff is the registry, CreateContainerConfigError is a missing ConfigMap or Secret, and CrashLoopBackOff is your process exiting. They share a symptom and nothing else, so the first job in any Kubernetes incident is to read the exact state rather than the summary.
The second recurring trap is that the most useful evidence expires. Events are garbage-collected after about an hour by default, so a failure that happened overnight leaves a pod in a broken state with no events to explain it. And in a crash loop, the *current* container has usually printed nothing — the stack trace you need belongs to the previous one, which is why `--previous` is the single most valuable flag in kubectl.
Helm adds one more layer of indirection: what you wrote is templates, what Kubernetes received is rendered YAML, and they can differ in ways that are invisible in the templates. Render before you debug.
Triage order
Work down this list in order. Each step either finds the fault or rules out a whole class of cause — a negative result is progress, not a wasted step.
-
What is the pod’s actual state?
kubectl get pod <pod> -o widePending means nothing scheduled it. ImagePullBackOff means the node cannot fetch the image. CreateContainerConfigError means a referenced ConfigMap or Secret is missing. CrashLoopBackOff means it started and exited. These have four different fixes.
-
What do the events say — and are they still there?
kubectl describe pod <pod> | sed -n "/Events:/,$p"Scheduling failures, image pull errors, probe failures and volume mount problems all appear here with a specific reason. No events at all on an old pod usually means they have expired, not that nothing happened.
-
What did the container print before it died?
kubectl logs <pod> --previousIn a crash loop the running container has often produced no output yet. `--previous` reads the last terminated container, which is where the actual error lives. Skipping this step is the most common reason a crash loop looks inexplicable.
-
Was it killed, or did it exit on its own?
kubectl get pod <pod> -o jsonpath="{.status.containerStatuses[*].lastState.terminated.reason} {.status.containerStatuses[*].lastState.terminated.exitCode}"`OOMKilled` means the memory limit was hit — the kernel killed it and no application log will explain why. Exit code 1 is your process failing; 137 is SIGKILL; 143 is SIGTERM during a normal shutdown.
-
Does the Service actually have endpoints?
kubectl get endpoints <service>Empty endpoints has exactly two causes: the selector matches no pods, or the pods that match are not Ready. `kubectl get pods --show-labels` distinguishes them immediately. A Service with no endpoints is a black hole that returns connection refused.
-
For Helm: what was actually deployed?
helm get manifest <release> | kubectl diff -f -Compares what the release believes it deployed against what is live in the cluster. Drift here means something changed the resources out of band, and the next `helm upgrade` will revert it.
Diagnostic commands
Every command is labelled by what it can do to the system. Read-only commands are safe to run during an incident; the others are not, and are marked accordingly.
kubectl logs <pod> --previous --tail=200 Read the output of the container instance that just died.
How to read it: The only reliable place to find the cause of a crash loop. Empty output plus an OOMKilled reason means the process never got to log anything before the kernel stopped it.
kubectl get events --sort-by=.lastTimestamp -A | tail -40 See what the cluster has been complaining about most recently, across namespaces.
How to read it: Best first command when something is wrong but you do not yet know where. Note that events expire — an empty list is not evidence of health.
kubectl describe node <node> | sed -n "/Allocated resources/,/Events/p" Show what is actually requested on a node versus its capacity.
How to read it: Explains Pending pods. Scheduling uses *requests*, not usage, so a node can be almost idle and still unschedulable if requests are fully committed.
kubectl get pods --show-labels | grep <selector-value> Verify that a Service selector matches real pods.
How to read it: If nothing matches, the selector is wrong. If pods match but endpoints are empty, they are failing readiness — two different fixes for one symptom.
kubectl debug -it <pod> --image=busybox --target=<container> Attach a debug container sharing the target’s process namespace.
How to read it: The correct way to inspect a distroless or minimal image. Adding a shell to the production image to make it debuggable enlarges the attack surface permanently.
kubectl rollout undo deployment/<name> Return a Deployment to its previous ReplicaSet.
How to read it: Fast and reliable for the workload, but it does not roll back anything the application did to a database. Check for migrations before relying on it.
kubectl delete pod <pod> --grace-period=0 --force Remove a pod stuck in Terminating.
How to read it: Deletes the API object without confirming the container stopped. On a StatefulSet with attached storage this can produce two writers on one volume — verify the node is genuinely gone first.
Failure modes
These are distinct problems, not variations of one. Matching the symptom to the right cause is most of the work.
CrashLoopBackOff with no useful logs.
- Cause:
- The current container has not produced output yet, or the process is being killed before it logs.
- Fix:
- Read `--previous` logs. If those are empty too, check the termination reason: OOMKilled means the memory limit is too low, and no application log will ever explain it.
Pod stays Pending indefinitely.
- Cause:
- No node satisfies its requirements — insufficient uncommitted requests, an unsatisfiable nodeSelector or affinity rule, or an unbound PVC.
- Fix:
- Read the scheduling event; it names the reason per node. A Pending PVC is often *correct*: with `WaitForFirstConsumer`, the volume is deliberately not provisioned until a pod is scheduled.
Service returns connection refused although pods are Running.
- Cause:
- The Service has no endpoints — selector mismatch, or pods running but not Ready.
- Fix:
- Check endpoints, then labels. Running and Ready are different states, and only Ready pods receive traffic.
The application is slow but nothing appears unhealthy.
- Cause:
- CPU throttling against the limit. CPU limits throttle rather than kill, so the workload degrades silently.
- Fix:
- Check `container_cpu_cfs_throttled_seconds_total`. Memory limits kill and are visible as OOMKilled; CPU limits produce latency with no error anywhere. This distinction explains most "slow but healthy" reports.
Helm upgrade fails and leaves the release in pending-upgrade.
- Cause:
- The previous operation was interrupted, so the release lock was never cleared.
- Fix:
- Inspect `helm history`, then roll back to the last known-good revision. Deleting the release secret by hand loses the history that makes future rollbacks possible.
ImagePullBackOff on some nodes only.
- Cause:
- The image pull secret is missing in that namespace, or those nodes lack registry credentials or network egress.
- Fix:
- Read the event — it distinguishes authentication failure from name resolution from connection timeout, and those are three different fixes.
Common mistakes
- Reading current logs in a crash loop instead of `--previous`, then concluding the container produces no output.
- Treating an empty event list as healthy. Events expire after roughly an hour by default.
- Setting CPU limits equal to requests on latency-sensitive services, guaranteeing throttling under normal bursts.
- Force-deleting pods stuck in Terminating without confirming the node is gone — on stateful workloads this risks two writers on one volume.
- Debugging rendered Helm output by reading the templates. Render first with `helm template`; the bug is usually visible in the output.
- Using `latest` tags, which makes it impossible to tell which build is running and defeats rollback.
Frequently asked questions
Why is my pod in CrashLoopBackOff with empty logs?
Because you are reading the wrong container. In a crash loop the running instance has typically not produced output yet; the evidence belongs to the previous one, which `kubectl logs --previous` reads. If that is empty too, check the termination reason — `OOMKilled` means the kernel stopped the process before it could log anything, and the fix is the memory limit rather than the application.
My pods are Running but the Service returns connection refused. Why?
The Service almost certainly has no endpoints, and there are exactly two causes: the selector matches no pods, or the pods it matches are not Ready. `kubectl get endpoints` confirms the condition and `kubectl get pods --show-labels` tells you which of the two it is. Running and Ready are separate states, and only Ready pods are sent traffic.
What is the difference between a CPU limit and a memory limit?
They fail in completely different ways. Exceeding a memory limit gets the container killed, which is visible and loud. Exceeding a CPU limit gets it throttled, which is silent — the workload just becomes slow while every health check continues to pass. That asymmetry is why "slow but nothing is wrong" so often turns out to be CPU throttling.
Is a Pending PersistentVolumeClaim always a problem?
No. With a `WaitForFirstConsumer` binding mode the volume is deliberately not provisioned until a pod that uses it is scheduled, so a Pending PVC with no pod is the design working as intended. It becomes a problem when a pod is scheduled and the claim still does not bind — then look at the storage class and provisioner.
How do I debug a container built from a distroless image?
Use `kubectl debug` to attach an ephemeral container that shares the target’s process namespace, which gives you a shell and tooling without changing the image. The alternative — adding a shell to the production image so it can be debugged — permanently enlarges the attack surface to solve an occasional problem.