Kubernetes Node Cordon, Drain & Maintenance Runbook Prompt
Produce a safe, repeatable runbook for taking a node out of service for patching or hardware work, respecting PodDisruptionBudgets, local storage, and DaemonSets.
- Target user
- On-call engineers and SREs performing node maintenance
- Difficulty
- Beginner
- Tools
- Claude, ChatGPT
The prompt
You are a careful SRE who has both drained nodes uneventfully and watched a single careless `--force` evict a stateful pod into data loss. I will provide: - Cluster size, node role (control-plane vs worker), and managed vs self-managed - Workloads on the node (stateless, StatefulSets, DaemonSets, pods with local storage) - Whether PodDisruptionBudgets exist and the maintenance window Your job: 1. **Pre-flight** — capture current state: `kubectl get pods -o wide --field-selector spec.nodeName=<node>`, check PDBs, and confirm enough spare capacity elsewhere to host evicted pods. 2. **Cordon first** — `kubectl cordon <node>` to stop new scheduling, and explain why cordon-then-observe is safer than cordon+drain in one motion. 3. **Drain correctly** — run `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data` and explain each flag. Discuss when `--force` is acceptable (only for unmanaged standalone pods you accept losing) and when it is dangerous. 4. **Respect PDBs** — interpret a drain that stalls on `Cannot evict pod as it would violate the budget`; the fix is more replicas or a wider window, not bypassing the PDB. 5. **Local storage caveat** — flag pods using `emptyDir` or local PVs: draining destroys emptyDir data and local-PV pods cannot reschedule. Decide per-pod. 6. **Maintenance & return** — do the work, then `kubectl uncordon <node>`, verify it goes Ready and re-receives pods, and confirm no workloads stuck Pending. Output as: (a) the ordered command runbook with verification after each step, (b) a decision table for force/local-storage cases, (c) the abort/back-out procedure (uncordon, investigate) if drain stalls. Never add `--force` reflexively to clear a stalled drain — a stall usually means a PDB or local-storage pod is correctly protecting state.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Helm Release History Pruning and max-history Prompt
Stop Helm release history from bloating etcd with hundreds of Secret/ConfigMap revisions — set --history-max, prune safely, and avoid hitting the 1MB Secret size limit or slow upgrades.
-
PVC Online Volume Expansion Runbook Prompt
Safely grow a PersistentVolumeClaim in place — verify the StorageClass allows expansion, patch the request, and confirm the filesystem actually resized without corrupting data or getting stuck in FileSystemResizePending.
-
Zero-Downtime Secret Rotation Runbook Prompt
Rotate a Kubernetes Secret (DB password, API key, TLS cert) without dropping traffic — reason about mounted-volume vs envFrom propagation, dual-key overlap windows, and forcing a controlled rollout only where needed.
-
Kubernetes PDB unhealthyPodEvictionPolicy Prompt
Stop PodDisruptionBudgets from deadlocking node drains when pods are already broken, by choosing the right unhealthyPodEvictionPolicy and minAvailable/maxUnavailable math.
More Kubernetes & Helm prompts & error guides
Browse every Kubernetes & Helm prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.