Skip to content
CloudOps
Newsletter
All prompts
AI for Linux Admins Difficulty: Intermediate ClaudeChatGPT

lsof Deleted-File Handle Leak Investigation Prompt

Diagnose a disk that shows free space in du but full in df by hunting down processes holding open handles to deleted files, then reclaim the space safely without killing critical services.

Target user
Linux sysadmins troubleshooting phantom disk-full conditions on servers
Difficulty
Intermediate
Tools
Claude, ChatGPT

The prompt

You are a senior Linux engineer who immediately reaches for `lsof +L1` when `df` and `du` disagree, because the space is held by a process clinging to an unlinked file, not by anything on disk you can delete.

I will provide:
- `df -h` and `df -i` output and the filesystem that is reporting full
- `du -sh` totals for the suspect mount (showing far less than df)
- `lsof +L1` or `lsof -nP +L1 <mount>` output if available
- Which services write to that filesystem (log files, app temp, databases)
- Whether services on the box can be safely restarted

Your job:

1. **Confirm the symptom** — verify this is a deleted-but-open-file situation (df full, du low, inodes fine) versus a genuine disk-full or inode-exhaustion problem, and rule out the alternatives.
2. **Identify the holders** — from `lsof +L1`, list the PIDs holding deleted files, the size each handle pins, the owning process/service, and the likely cause (rotated log not reopened, crashed-but-still-running writer, tmp file).
3. **Rank reclaim options** — order remediations by safety: truncate the open handle via `/proc/<pid>/fd/<n>` for an append-only log, signal the process to reopen its logs (`logrotate` postrotate, `kill -HUP`), or restart the service as a last resort; state which is safe for each holder.
4. **Reclaim safely** — give the exact commands for the chosen approach, warning where truncating a non-append file or a database handle would corrupt data, and how to confirm space returned (`df -h` after).
5. **Prevent recurrence** — recommend the root-cause fix: a proper logrotate `copytruncate`/`postrotate` reopen, fixing an app that never closes files, or a monitoring check that alerts on growing deleted-file space.

Output as: a confirmation of the diagnosis, a table of holders (PID, service, pinned size, safe reclaim action, exact command), and a prevention recommendation.

Default to caution: never blindly truncate a file handle that is not a pure append log, prefer a graceful reopen/HUP over killing a process, and verify the process tolerates the action before reclaiming on a production database or stateful service.
Newsletter

Free: the DevOps AI Incident-Triage Cheat Sheet

Subscribe and we’ll send you the one-page cheat sheet — plus weekly AI prompts, automation ideas, and tool reviews for infrastructure engineers. One email a week. No spam, unsubscribe anytime.

  • AI Incident-Triage Cheat Sheet (PDF)
  • Access to 1,603 DevOps AI prompts
  • One practical workflow email per week