`kubectl debug` & Ephemeral Containers Prompt

You are a senior Kubernetes engineer who uses `kubectl debug` daily to diagnose production pods without restarting them. You know the three modes (target an existing pod, debug a node, copy a pod) and the common gotchas with shared namespaces and security contexts.

I will provide:
- The pod or node to debug
- The symptom (distroless pod with no shell, broken pod that won't restart, network issue from inside the pod, node-level investigation)
- The K8s version (ephemeral containers GA in 1.25)
- Cluster security posture (PSA labels, admission controllers blocking elevated containers)

Your job:

1. **Pick the right `kubectl debug` mode**:
   - **`kubectl debug <pod> --image=<debug-image>`** — attach an ephemeral container to a running pod; shares pod's network and (if requested) PID namespace
   - **`kubectl debug <pod> --copy-to=<new-pod> --share-processes`** — copy the pod with modifications (different image, env, etc.) — useful when the pod is crash-looping
   - **`kubectl debug node/<node>`** — pod that runs on a node with host filesystem mounted; useful for node-level inspection
2. **For attaching to a running pod**:
   - `kubectl debug <pod> -it --image=nicolaka/netshoot` — drops you into netshoot
   - `--target=<container>` shares process namespace with that container (lets you see its processes)
   - For distroless / scratch pods (no shell), ephemeral container brings tools without modifying the image
3. **For debugging crashed pods (`copy-to` mode)**:
   - `kubectl debug <pod> -it --copy-to=<pod>-debug --image=alpine -- sh`
   - Creates a NEW pod from the crashed one's spec with modifications
   - `--share-processes` lets the debug container see the original's processes
   - Useful when the pod CrashLoops and you can't `exec` in
4. **For node-level debugging**:
   - `kubectl debug node/<node-name> -it --image=nicolaka/netshoot`
   - Creates a pod running on that node with the node's filesystem mounted at `/host`
   - `chroot /host` to act as if you're on the node
   - Requires sufficient RBAC (create pods + privileged)
5. **For PSA / admission restrictions**:
   - Debug containers run with the pod's security context unless overridden
   - `Restricted` PSA disallows privileged debug containers in workload namespaces
   - Workaround: run debug pod in a permissive namespace, share network via service mesh, etc.
6. **Common debug images**:
   - `nicolaka/netshoot` — network tools (dig, tcpdump, ss, mtr, curl, jq)
   - `busybox` — basic
   - `ubuntu` — apt-installable extras
   - `alpine` — apk-installable
   - `cilium/cilium-cli` — for Cilium-specific
7. **For inspecting a pod's filesystem**:
   - `kubectl debug <pod> -it --image=alpine --target=<container>` shares the container's filesystem via `/proc/<pid>/root`
   - Or `kubectl cp <pod>:/path /tmp/local` to copy out files

Mark DESTRUCTIVE: debug commands that write to the target pod's filesystem (changes persist), `kubectl debug --replace` (replaces the pod), running privileged debug pods in production without auditing.

---

Target: [pod / node / crashed pod]
Symptom + debug goal: [DESCRIBE]
K8s version: [DESCRIBE]
PSA labels on namespace: [`baseline` / `restricted` / `privileged`]
What you've already tried:
[DESCRIBE]

Why this prompt works

kubectl debug removes the “I can’t exec into this pod because it has no shell” problem and enables non-disruptive debugging. Most engineers know about it but don’t use the target/copy-to/node modes. This prompt picks the right mode per scenario.

How to use it

1. Pick the mode by scenario: attach for running pods, copy-to for crashed ones, node/ for node-level work.
2. Prefer netshoot for network debugging — it has the tools you’d want.
3. For distroless workloads, ephemeral container is the right answer.
4. Audit privileged debug pod usage in production.

Useful commands

# Attach ephemeral container to a running pod
kubectl debug -it <pod> --image=nicolaka/netshoot

# Share process namespace with a specific container
kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container-name>

# Specify image and just run a command
kubectl debug <pod> --image=alpine -- ls /

# Copy a crashed pod and modify
kubectl debug <pod> -it \
    --copy-to=<pod>-debug \
    --share-processes \
    --image=ubuntu \
    -- bash

# Override the entrypoint to keep the copy running
kubectl debug <pod> -it \
    --copy-to=<pod>-debug \
    --container=<container> \
    --image=ubuntu \
    --set-image=*=ubuntu \
    -- sleep 3600

# Node-level debug
kubectl debug node/<node-name> -it --image=nicolaka/netshoot
# Inside, chroot to host
# chroot /host

# View ephemeral containers on a pod
kubectl get pod <pod> -o jsonpath='{.spec.ephemeralContainers[*].name}'

# Logs from ephemeral container
kubectl logs <pod> -c <ephemeral-container-name>

Common scenarios

Distroless pod, no shell, network issue

kubectl debug -it <distroless-pod> --image=nicolaka/netshoot --target=app
# Now inside netshoot, same network as the distroless pod:
# curl http://internal-service:8080/healthz
# dig internal-service
# ss -tnp

Crashed pod, can’t exec, want to inspect filesystem

# Copy the crashed pod with a different entrypoint
kubectl debug <crashing-pod> -it \
    --copy-to=<crashing-pod>-debug \
    --container=<container-name> \
    --set-image=*=ubuntu:22.04 \
    -- bash

# Inside, the original container's PVCs are mounted at the same paths
# Inspect logs, config files, lockfiles, etc.

# Clean up
kubectl delete pod <crashing-pod>-debug

Network policy debugging from inside a specific pod

kubectl debug -it <pod> --image=nicolaka/netshoot
# nc -vz target-service 80
# tcpdump -i eth0 host target-ip
# ss -tnp

Node-level filesystem inspection

kubectl debug node/<node-name> -it --image=nicolaka/netshoot
chroot /host
df -h
ls /var/lib/containerd/
journalctl -u kubelet --since "1 hour ago"

Look at the actual filesystem the container sees

kubectl debug -it <pod> --image=alpine --target=<container>
# Inside, the target container's filesystem is at /proc/<pid>/root
ls /proc/1/root/etc/
cat /proc/1/root/etc/myapp.conf

Common findings this catches

• “No shell in distroless” → use kubectl debug with netshoot.
• CrashLoopBackOff → copy-to mode with sleep entrypoint to inspect.
• NetworkPolicy block → debug from a pod ON that namespace to verify reach.
• Node disk pressure → kubectl debug node/ then chroot and df -h.
• Secret content → kubectl debug attaches; can cat mounted secrets (audit access).
• kubelet logs from inside cluster → kubectl debug node/ → chroot /host → journalctl -u kubelet.

Cleanup

# Ephemeral container goes away with pod
kubectl delete pod <pod>          # also removes ephemeral debug container

# Copy-to pod must be deleted manually
kubectl delete pod <pod>-debug

# Node debug pod
# (visible in kube-system if no namespace specified)
kubectl get pods -A | grep debug
kubectl delete pod <node-debug-pod>

When to escalate

• Persistent need for privileged debug pods → consider building a secure debug-as-a-service for the team.
• Production secrets exposed through debug sessions → review RBAC; debug pods inherit privilege.
• PSA blocking debug → coordinate with security team for an exception process.