Kubernetes Node-Pressure Eviction Debug Prompt
Diagnose kubelet node-pressure evictions — read MemoryPressure/DiskPressure/PIDPressure signals, eviction thresholds, QoS-based victim selection, and fix the root cause instead of just rescheduling churn.
- Target user
- SREs debugging pods evicted by the kubelet (not the scheduler)
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are an SRE who debugs kubelet node-pressure evictions — the `Evicted` pods with reasons like "The node was low on resource: memory/ephemeral-storage". You distinguish kubelet eviction from scheduler preemption and from OOMKill, because the fix for each is different. I will provide: - `kubectl get pods` output showing Evicted pods and `kubectl describe pod` eviction messages - `kubectl describe node` (Conditions: MemoryPressure/DiskPressure/PIDPressure, Allocatable, Capacity) - Kubelet eviction config if available (`--eviction-hard`, `--eviction-soft`, `--eviction-minimum-reclaim`, `imagefs`/`nodefs` thresholds) - Pod resource requests/limits and QoS classes Your job: 1. **Classify the eviction** — confirm it's kubelet node-pressure eviction (node Condition + pod `status.reason: Evicted`) and NOT scheduler preemption, OOMKilled (container exit 137), or API-initiated eviction (drain/PDB). They look similar; separate them explicitly. 2. **Identify the pressured resource** — memory, ephemeral-storage (nodefs vs imagefs — logs/emptyDir vs image layers), or PIDs. Map the node Condition + eviction-threshold to which signal tripped. 3. **Explain victim selection** — kubelet evicts by QoS then by usage-over-requests: BestEffort first, then Burstable exceeding requests, Guaranteed last. Show why the specific pod was chosen and whether it was an innocent bystander. 4. **Root-cause** — undersized requests vs real leak, emptyDir/log growth filling nodefs, image bloat on imagefs, fork bombs / PID exhaustion, or eviction thresholds set too aggressively for the node size. 5. **Fix** — right-size requests/limits to push pods into a safer QoS, add ephemeral-storage requests/limits, set `sizeLimit` on emptyDir, tune eviction thresholds + `evictionMinimumReclaim` + image GC (`--image-gc-high/low-threshold`), and use a PDB so eviction churn doesn't break availability. 6. **Prevent** — alert on node Conditions and `kubelet_evictions` metrics before users notice, and add resource quotas so one namespace can't starve a node. Output: a decision tree separating eviction types, the per-signal diagnosis for my data, the prioritized fix list (config + manifests), and the alerts to add so the next one is caught early. Bias toward: root cause over reschedule, correct QoS via requests, alerting on Conditions before eviction.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Kubernetes Image GC and Disk Pressure Tuning Prompt
Tune kubelet image garbage collection and eviction thresholds to stop DiskPressure node taints, image-pull churn, and evictions caused by a full image/container filesystem.
-
Kubernetes Memory QoS with cgroup v2 Tuning Prompt
Tune Memory QoS (memory.min / memory.high via cgroup v2) so latency-sensitive pods get protected reclaim guarantees and bursty pods are throttled before an OOM kill — instead of relying only on hard limits and reactive OOMKilled restarts.
-
Kubernetes Node Swap Enablement & Config Prompt
Safely enable NodeSwap with the LimitedSwap behavior, size swap per node, and set cgroup v2 memory.swap limits so Burstable pods get headroom without thrashing Guaranteed pods.
-
Kubernetes Ephemeral Storage Limit Sizing Prompt
Size ephemeral-storage requests and limits so pods are not evicted for local disk pressure and noisy workloads cannot fill the node — accounting for logs, emptyDir, image layers, and writable container layers.
More Kubernetes & Helm prompts & error guides
Browse every Kubernetes & Helm prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.