Kubernetes Commands

Kubernetes tells you almost everything you need in two commands — kubectl describe and kubectl get events — and most wasted debugging time comes from running them in the wrong order, or from misreading which layer failed. A pod that will not start is a scheduling problem, an image problem, a config problem or a process problem, and those four have nothing in common except the symptom. This reference is organized around telling them apart quickly.

kubectl essentials

The triage order

Almost every investigation follows the same four steps. Doing them in this order is what makes it fast.

# 1. Are the NODES healthy? A NotReady node explains every pod on it.
kubectl get nodes -o wide

# 2. What is not running?
kubectl get pods -A --field-selector=status.phase!=Running -o wide

# 3. WHY — the events explain the scheduler's and kubelet's decisions
kubectl describe pod <pod> -n <ns>          # the Events section at the bottom
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -30

# 4. What did the process actually say?
kubectl logs <pod> -n <ns> --tail=100
kubectl logs <pod> -n <ns> --previous       # the instance that CRASHED

Pod startup failures: four different problems

The status field is diagnostic. These four are not variations of one issue.

# Exit code and OOM flag for every container in a pod
kubectl get pod <pod> -n <ns> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.lastState.terminated.reason}{"\n"}{end}'

Exit codes worth recognising: 137 is SIGKILL, and with reason: OOMKilled it means the container exceeded its memory limit. 143 is SIGTERM, a normal shutdown. 1 or 2 is usually the application failing on its own terms — read the logs, not the cluster.

Guides: CrashLoopBackOff, ImagePullBackOff, ErrImagePull, CreateContainerConfigError, ConfigMap not found, OOMKilled, pod stuck Pending and pod stuck Terminating.

Scheduling: why a pod stays Pending

kubectl describe pod names the reason directly — the scheduler explains itself in the events.

kubectl describe pod <pod> -n <ns> | grep -A15 Events
# "0/5 nodes are available: 3 Insufficient cpu, 2 node(s) had untolerated taint..."

That message is a complete answer if you read it as a tally: it says how many nodes were rejected and why. The usual causes:

• Insufficient cpu/memory — the pod’s requests do not fit on any node. Requests, not limits, drive scheduling.
• Untolerated taint — the node is marked for something else and the pod has no matching toleration.
• Node affinity / selector — no node carries the required labels.
• Unbound PersistentVolumeClaim — the volume cannot be provisioned, so the pod cannot be placed.
• Exceeded quota — a ResourceQuota in the namespace rejected it before scheduling.

# What is actually allocatable, and what is already requested?
kubectl describe node <node> | grep -A8 "Allocated resources"

# Taints on every node, in one line each
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Namespace quota usage
kubectl describe resourcequota -n <ns>

See FailedScheduling, untolerated taint / node affinity, exceeded quota and volume node affinity conflict.

Resources, limits and eviction

Quality of Service class decides who gets evicted first when a node runs short:

Guaranteed   requests == limits for every container   evicted last
Burstable    requests set, limits higher or unset     evicted in the middle
BestEffort   no requests or limits at all             evicted FIRST

kubectl get pod <pod> -n <ns> -o jsonpath='{.status.qosClass}'
kubectl top nodes                 # needs metrics-server
kubectl top pods -n <ns> --sort-by=memory

Anything critical should be Burstable at minimum, with memory requests set honestly — a BestEffort pod is the first thing the kubelet removes under pressure. Node-level pressure produces evicted: node was low on resource, node has disk pressure, memory pressure and ephemeral storage exceeded. If kubectl top returns nothing at all, metrics-server is missing — metrics.k8s.io not available.

Services, endpoints and DNS

A Service that resolves but never answers is nearly always the same bug.

# THE command. An empty endpoint list means the selector matches no READY pod.
kubectl get endpoints <svc> -n <ns>
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>

# Compare the selector against the pods' actual labels
kubectl get svc <svc> -n <ns> -o jsonpath='{.spec.selector}'; echo
kubectl get pods -n <ns> --show-labels

# Test DNS from inside the cluster
kubectl run dnstest --rm -it --restart=Never --image=busybox:1.36 -- \
  nslookup <svc>.<ns>.svc.cluster.local

Readiness and liveness are also frequently confused: a failing readiness probe removes the pod from the Service’s endpoints, so traffic stops but the container keeps running. A failing liveness probe restarts the container. Using a liveness probe where you meant readiness turns a slow dependency into a restart loop — and a startup probe exists precisely to stop liveness from killing an application that legitimately takes a while to boot.

Guides: no endpoints available for service, liveness probe failed / connection refused, CoreDNS SERVFAIL, connection refused in-cluster, ingress 503 and ingress 413.

Storage

kubectl get pvc -A | grep -v Bound          # anything not Bound blocks a pod
kubectl describe pvc <pvc> -n <ns>          # the provisioner explains itself here
kubectl get storageclass                    # is there a (default) one?

The genuine failures are a missing default StorageClass, an access mode the driver does not support (ReadWriteMany on a block-storage provisioner), or a zone mismatch between the volume and the node. See PersistentVolumeClaim not bound, failed to provision volume, FailedMount, MountVolume.SetUp failed, multi-attach error and fsGroup permission denied.

Deployments, rollouts and probes

ProgressDeadlineExceeded (600 seconds by default) means the rollout did not complete in time — the new pods never became Ready. The Deployment is reporting a symptom; the cause is in the new pods, so go back to the triage order. See ProgressDeadlineExceeded, failed to create ReplicaSet and field is immutable — the last means you changed something a Deployment cannot update in place, such as a selector.

Jobs and CronJobs

kubectl get jobs -n <ns>
kubectl describe job <job> -n <ns>

# The Job is the symptom — the POD holds the reason
kubectl logs job/<job> -n <ns> --all-containers --tail=100
kubectl get pods -n <ns> --selector=job-name=<job> --show-labels

BackoffLimitExceeded means the pod kept failing until the Job ran out of retries (backoffLimit, default 6). The Job status tells you nothing about why — read the last failed pod’s logs. Note also that a Job’s restartPolicy must be Never or OnFailure; Always is rejected. See BackoffLimitExceeded.

Helm

A release stuck in pending-upgrade usually means a previous helm upgrade was interrupted — its lock is still held, and the next attempt fails with “another operation is in progress”. Roll back to the last good revision rather than forcing it. See Helm upgrade failed, another operation in progress and rendered manifests contain a resource that already exists.

Debugging when exec is not enough

Cluster-plane failures worth knowing: FailedCreatePodSandbox and CNI config uninitialized are CNI problems, failed calling webhook means an admission webhook is unreachable — and with failurePolicy: Fail that blocks every matching create in the cluster — and context deadline exceeded is a timeout against the API server or a webhook, not a crash.

Troubleshooting specific errors

The Kubernetes failures engineers hit most often, each with a dedicated guide:

• BackoffLimitExceeded — the Job ran out of retries; read the pod, not the Job.
• context deadline exceeded — a timeout: slow API server, etcd, or an admission webhook.
• PersistentVolumeClaim not bound — StorageClass, access mode, or WaitForFirstConsumer.
• CrashLoopBackOff — the process exits; --previous logs hold the reason.
• ImagePullBackOff — name, tag, registry auth or network.
• FailedScheduling — read the node tally in the event.
• OOMKilled — exit 137; the memory limit is a hard wall.
• pod stuck Terminating — usually a finalizer.
• ProgressDeadlineExceeded — the new pods never went Ready.
• FailedCreatePodSandbox — CNI could not set up networking.
• failed calling webhook — an admission webhook is down.
• no endpoints available for service — selector mismatch, or pods not Ready.
• NodeNotReady and node is unreachable — start here before debugging pods.
• Helm upgrade failed — check helm history, then roll back.
• annotations too long / 262144 — the last-applied-configuration annotation; use server-side apply.

For anything else, browse the Kubernetes error clusters or paste the message into the Incident Assistant.

Production checklist

• Check the context before anything destructive. Put it in your shell prompt.
• Nodes first, then pods, then events, then logs. A NotReady node explains everything on it.
• kubectl logs --previous in a crash loop. The current container has not said anything yet.
• Set memory requests honestly. BestEffort pods are evicted first.
• Remember CPU limits throttle and memory limits kill — they are not the same control.
• Readiness removes from endpoints; liveness restarts. Use a startup probe for slow boots.
• kubectl get endpoints when a Service does not answer. Empty means selector or readiness.
• kubectl diff before kubectl apply.
• --atomic on helm upgrade in CI, so a failure rolls itself back.
• kubectl debug for distroless images instead of weakening them to add a shell.
• Know that events expire. An hour-old failure may leave none.
• Keep kubectl within one minor version of the API server.

Frequently asked questions

What is the difference between Pending and CrashLoopBackOff?

They are different layers. Pending means the pod has not been scheduled onto a node at all — the scheduler could not find somewhere to put it, because of resource requests, taints, node affinity or an unbound volume. The container has never run. CrashLoopBackOff means the opposite: the pod was scheduled, the image was pulled, the container started, and the process exited — repeatedly, so the kubelet is backing off between restarts. For Pending you read kubectl describe pod events; for CrashLoopBackOff you read kubectl logs --previous.

Why is my Service not reaching any pods?

Run kubectl get endpoints <svc>. An empty list has exactly two causes: the Service’s selector does not match any pod’s labels, or the pods match but are not Ready because a readiness probe is failing. kubectl get pods distinguishes them — pods showing 0/1 Running mean the labels are fine and the probe is the problem. A Service with populated endpoints that still does not answer is a network policy, a port mismatch, or the application not listening on the address you think.

What does exit code 137 mean?

The container received SIGKILL. Combined with reason: OOMKilled in kubectl describe pod, it means the container exceeded its memory limit and the kernel killed it — the limit is a hard wall, not a target. Raise the limit if the usage is legitimate, or fix the leak if it is not; and set the memory request too, since that is what the scheduler uses. Exit 137 without OOMKilled usually means something else sent SIGKILL, such as a node drain that ran past the termination grace period.

Should I use a liveness or a readiness probe?

Readiness controls traffic; liveness controls restarts. A failing readiness probe removes the pod from the Service’s endpoints while leaving it running, which is what you want when a dependency is briefly unavailable. A failing liveness probe restarts the container, which is what you want only when the process is genuinely wedged. Using liveness where you meant readiness turns a slow database into a cluster-wide restart storm. For an application that takes a while to boot, add a startup probe so liveness does not kill it before it is ready.

How do I debug a container with no shell?

Use kubectl debug -it <pod> --image=busybox:1.36 --target=<container>. It attaches an ephemeral container to the running pod, sharing its process and network namespaces, so you get your own tooling against the real process without rebuilding the image or adding a shell to a hardened one. For node-level problems, kubectl debug node/<node> gives you a privileged pod with the host filesystem mounted at /host.

Why does my pod stay Terminating forever?

Almost always a finalizer: some controller registered itself to run cleanup before the object can be removed, and it is not completing — because the controller is gone, unhealthy, or waiting on something. Check kubectl get pod <pod> -o yaml | grep -A5 finalizers. The other cause is an unresponsive kubelet on that node, in which case the API server cannot confirm the pod is gone. Force-deleting with --grace-period=0 --force removes the API object but does not stop a still-running container, so it should be a last resort rather than a habit.

Related resources

• Guide: Kubernetes Security — RBAC, admission control, network policy and workload hardening.
• Guide: Docker Compose — the container fundamentals underneath every pod.
• Guide: Linux Commands — node-level diagnosis when the problem is below Kubernetes.
• Stack hub: Kubernetes command centre — the top Kubernetes errors, tools and runbook.
• Troubleshooting hubs: pod startup, networking and ingress and storage error clusters.
• Tool: Incident Assistant — paste an error and get an ordered triage plan.