🔬 Debugging Distroless Pods With Ephemeral Debug Containers

You finally hardened your images — distroless, no shell, no package manager, minimal attack surface. Then a pod misbehaves at 2am, you reach for kubectl exec -it pod -- sh, and you get exec: "sh": executable file not found. There’s nothing in the container to debug with. The old answer was to bake busybox into production images, which throws away the hardening you just did. The right answer is ephemeral debug containers: attach a throwaway container with all your tools into the running pod’s namespaces, debug, then let it vanish. This is the technique that lets you keep minimal images and stay debuggable.

What an ephemeral container is

An ephemeral container is a temporary container you inject into an existing pod. It shares the pod’s network namespace (so it sees the same localhost and interfaces) and, when you ask, the process namespace of a target container (so it can see and inspect that container’s processes and filesystem). It has no restart policy, no resources guarantees, and it disappears when you’re done. Crucially, it doesn’t change the pod spec or trigger a restart — your broken pod stays exactly as broken as it was, so you can observe the real failure state.

kubectl debug, the everyday case

The command is kubectl debug. To drop a full-featured toolbox into a running pod:

kubectl debug -it payments-7d9f-abcde \
  --image=nicolaka/netshoot \
  --target=payments

• --image is your debug image. netshoot is the go-to — it bundles curl, dig, tcpdump, ss, nslookup, iperf, and most of what you’d want for network triage.
• --target=payments shares the process namespace of the payments container, so you can see its processes and, via /proc, its filesystem and environment.
• -it gives you an interactive shell.

Now you’re inside the pod’s network, with real tools, looking at the broken container’s processes — and your production image still has no shell baked in.

What you can actually do once you’re in

This is where it pays off. From the debug container you can:

# Is DNS resolving from inside this pod's network?
nslookup payments-db.payments.svc.cluster.local

# Can this pod actually reach the dependency?
curl -v http://payments-db:5432

# What's the target container doing right now?
ps aux

# Capture traffic on the shared network namespace
tcpdump -i any port 5432 -c 20

That DNS-and-connectivity pair answers the most common “service can’t talk to its dependency” question in about thirty seconds — and it answers it from inside the actual pod’s network, which is the only place the answer is trustworthy. Testing connectivity from your laptop or another namespace can lie because of NetworkPolicy or service-mesh rules.

Inspecting the target container’s filesystem

With --target, the debug container shares the PID namespace, so the target’s root filesystem is reachable through /proc:

ls -l /proc/1/root/app/config/
cat /proc/1/root/etc/resolv.conf

PID 1 in the shared namespace is usually the target’s main process, and /proc/1/root is its filesystem root. This is how you read the distroless container’s config files when it has no cat of its own.

Debugging a pod that won’t start

Ephemeral containers attach to a running pod. If the pod is in CrashLoopBackOff, there’s no running process to attach to. For that, kubectl debug can create a copy of the pod with the command changed, so you get a debuggable version without the crashing entrypoint:

kubectl debug payments-7d9f-abcde -it \
  --image=nicolaka/netshoot \
  --copy-to=payments-debug \
  --share-processes \
  -- sh

This spins up payments-debug alongside the broken pod, with a shell instead of the failing command, so you can poke at the environment, mounts, and config that the real pod would have seen.

Debugging a node, not a pod

kubectl debug also targets nodes. This gives you a privileged container with the node’s root filesystem mounted at /host — useful when the problem is the node, not the workload:

kubectl debug node/ip-10-0-1-23 -it --image=ubuntu
# inside: chroot /host to inspect kubelet logs, disk, etc.

Treat this as the powerful, scary tool it is — it’s effectively root on the node.

Operational notes

• Ephemeral containers must be enabled, but they’ve been on by default since Kubernetes 1.25. On very old clusters you may need a feature gate.
• RBAC governs this. Creating ephemeral containers requires the pods/ephemeralcontainers subresource permission. That’s good — you don’t want everyone able to inject privileged tooling into production pods. Scope it to your on-call role.
• It still respects security context. If the pod runs as non-root, your debug container may inherit constraints. For some debugging you’ll want a debug image and explicit security context that allow the tooling you need — within policy.
• Pin a known debug image in your runbook so you’re not picking one at 2am. netshoot for network, busybox for basic shell work.

The whole point is that you no longer trade hardening for debuggability. Keep your images distroless, keep your attack surface tiny, and attach the toolbox only when you need it — then watch it disappear.

If you maintain shared debug images or the RBAC that gates this access, it’s worth a review pass to make sure the ephemeral-container permission isn’t wider than your on-call group. Our AI code review is good at spotting over-broad RBAC subresource grants.

Ephemeral debug containers are the missing piece that makes minimal images practical in production. For more troubleshooting and operations guides, see the Kubernetes & Helm category.

Debug containers and node debugging grant powerful access. Verify RBAC scoping and security context against your own policies before relying on these in production.