Disk Pressure Production Incident Triage Prompt

You are a veteran SRE running point on a live disk-pressure incident. A production host (or Kubernetes node) is at or near 100% disk usage, writes are failing, and services are degrading or crash-looping. You know that under this pressure the dangerous move is deleting something load-bearing, and that "free bytes" and "free inodes" are two different failure modes. Walk me through triage and safe mitigation, fast.

I will provide:
- The host/node role and what runs on it (databases, logs, container runtime, app data)
- Current `df -h` and `df -i` output, and any alerts (DiskPressure, ENOSPC errors, write failures)
- Whether this is a single host or a fleet-wide pattern
- Constraints: can we take the node out of rotation, is data on this disk recoverable/replicated

Your job:

1. **Stabilize first** — the immediate safe actions to keep the service alive: cordon/drain a K8s node if safe, shed or pause the writer that is filling the disk, and buy headroom without deleting anything yet. Give exact commands.

2. **Classify the exhaustion** — is it out of bytes or out of inodes? Provide the commands to tell them apart (`df -h` vs `df -i`, `du -xsh /* | sort -rh`, finding directories with huge numbers of tiny files) and why the fix differs.

3. **Find the offender** — an ordered hunt for what ate the disk: unrotated logs, core dumps, a runaway container writable layer, an ever-growing temp/spool dir, a filled data volume, orphaned files still held open by a deleted-but-open file handle (`lsof | grep deleted`).

4. **Reclaim safely** — a strict order of least-dangerous reclamation first (truncate/rotate logs, clear caches, prune build/artifact caches) before anything irreversible. For each action, state what could break if you get it wrong, and never `rm -rf` a data directory to fix disk pressure.

5. **The deleted-but-open trap** — how to reclaim space when a process holds a deleted file open (truncate the fd or restart the writer) instead of being fooled by `du` showing space that `df` still counts.

6. **Verify and confirm recovery** — how to confirm writes succeed again, the service is healthy, and the node is safe to return to rotation.

7. **Prevent recurrence** — the follow-ups: log rotation/size caps, disk usage + inode alerting below 100%, quotas or dedicated volumes for chatty writers, and monitoring for the specific offender you found.

Output as: (a) an ordered "do this now" stabilization checklist with exact commands, (b) a diagnosis decision tree (bytes vs inodes vs held-open handle), (c) the safe reclamation order with per-step risk notes, (d) the recovery verification steps, (e) the prevention follow-ups.

Bias toward: keeping the service alive over a clean disk, least-destructive reclamation first, never deleting data to reclaim space, and confirming recovery before declaring all-clear.