Kubernetes Production Troubleshooting

The failures that actually page you: pods that won’t start, networking and ingress, storage, and a security pass on the cluster.

0 of 4 modules complete

Who it’s for: Engineers operating Kubernetes who want a systematic triage path.
Prerequisites: Comfort with kubectl and core objects (Pod, Service, Deployment).

Skills you’ll build

✓Diagnose pod startup failures
✓Debug Service/Ingress networking
✓Resolve storage/PVC issues
✓Apply security guardrails

4 modules · self-paced

Module 1

Pod startup failures

Work CrashLoopBackOff, ImagePullBackOff, and pending pods to root cause.

Kubernetes pod startup errors hub
Diagnostic commands run in order — each one narrows the fault
1. Find nodes that are not ready
```
kubectl get nodes -o wide
```
  Start here. A NotReady node explains every pod problem scheduled onto it, and saves you debugging the wrong layer.
2. Find pods that are not running
```
kubectl get pods -A --field-selector=status.phase!=Running -o wide
```
  Pending means scheduling; CrashLoopBackOff means the process exits; ImagePullBackOff means registry or auth.
3. Read recent cluster events
```
kubectl get events -A --sort-by=.lastTimestamp | tail -40
```
  Events explain WHY the scheduler or kubelet made its decision — usually the fastest single command in Kubernetes triage.
4. Describe the failing pod
```
kubectl describe pod <POD> -n <NS>
```
  The Events section at the bottom, plus Last State / Reason, gives the exit code and OOMKilled flag.
5. Read the previous container log
```
kubectl logs <POD> -n <NS> --previous --tail=100
```
  `--previous` is the one that matters in a crash loop: it shows the log of the instance that actually died.
Exercise

A pod is stuck in CrashLoopBackOff. Use logs, describe, and events to determine whether it’s the image, config, probes, or resources.
Open in Workspace →
Module 2

Networking and ingress

Trace a request from Ingress → Service → Pod and find where it breaks.

Kubernetes networking & ingress errors hub
Diagnostic commands run in order — each one narrows the fault
1. Check the service has endpoints
```
kubectl get endpoints <SVC> -n <NS>
```
  An empty endpoint list means the selector matches no ready pods — the service is fine, the pods are not.
2. Verify selector and pod labels agree
```
kubectl get svc <SVC> -n <NS> -o jsonpath="{.spec.selector}" && echo && kubectl get pods -n <NS> --show-labels
```
  A single mismatched label is the most common cause of a service that resolves but never answers.
3. Test DNS from inside the cluster changes state
```
kubectl run dnstest --rm -it --restart=Never --image=busybox:1.36 -- nslookup <SVC>.<NS>.svc.cluster.local
```
  Failure here points at CoreDNS or the pod DNS policy, not at your application.
4. Check ingress routing
```
kubectl describe ingress <ING> -n <NS>
```
  Confirm the backend service and port, and check the events for a missing IngressClass or a TLS secret that does not exist.
Module 3

Storage and volumes

Resolve PVC binding, mount, and provisioning failures.

Kubernetes storage errors hub
Diagnostic commands run in order — each one narrows the fault
1. Find unbound volume claims
```
kubectl get pvc -A | grep -v Bound
```
  A Pending PVC blocks pod startup entirely. The reason is almost always a missing StorageClass or no matching PV.
2. Read the volume binding events
```
kubectl describe pvc <PVC> -n <NS>
```
  The events name the provisioner and its exact complaint — the fastest route to the real cause.
3. Check the default storage class
```
kubectl get storageclass
```
  No class marked `(default)` means every claim without an explicit className stays Pending forever.
Module 4

Cluster security pass

Apply the security guardrails that prevent a class of production incidents.

Kubernetes Security core guide Open the Troubleshooting Workspace
Diagnostic commands run in order — each one narrows the fault
1. Check what a service account can actually do
```
kubectl auth can-i --list --as=system:serviceaccount:<NS>:<SA> -n <NS>
```
  Read this as the blast radius of that pod if it is compromised. Wildcards on secrets or pods/exec deserve attention.
2. Find containers running as root
```
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" runAsNonRoot="}{.spec.securityContext.runAsNonRoot}{"\\n"}{end}' | grep -v "runAsNonRoot=true"
```
  Anything without runAsNonRoot=true is running as UID 0 unless the image says otherwise.
3. Find privileged containers
```
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[]?.securityContext.privileged == true) | .metadata.namespace + "/" + .metadata.name'
```
  A privileged container is effectively root on the node. Each one needs a specific, documented justification.

Browse the Kubernetes stack

Pod startup failures

Networking and ingress

Storage and volumes

Cluster security pass