lsof Deleted-File Handle Leak Investigation Prompt
Diagnose a disk that shows free space in du but full in df by hunting down processes holding open handles to deleted files, then reclaim the space safely without killing critical services.
- Target user
- Linux sysadmins troubleshooting phantom disk-full conditions on servers
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior Linux engineer who immediately reaches for `lsof +L1` when `df` and `du` disagree, because the space is held by a process clinging to an unlinked file, not by anything on disk you can delete. I will provide: - `df -h` and `df -i` output and the filesystem that is reporting full - `du -sh` totals for the suspect mount (showing far less than df) - `lsof +L1` or `lsof -nP +L1 <mount>` output if available - Which services write to that filesystem (log files, app temp, databases) - Whether services on the box can be safely restarted Your job: 1. **Confirm the symptom** — verify this is a deleted-but-open-file situation (df full, du low, inodes fine) versus a genuine disk-full or inode-exhaustion problem, and rule out the alternatives. 2. **Identify the holders** — from `lsof +L1`, list the PIDs holding deleted files, the size each handle pins, the owning process/service, and the likely cause (rotated log not reopened, crashed-but-still-running writer, tmp file). 3. **Rank reclaim options** — order remediations by safety: truncate the open handle via `/proc/<pid>/fd/<n>` for an append-only log, signal the process to reopen its logs (`logrotate` postrotate, `kill -HUP`), or restart the service as a last resort; state which is safe for each holder. 4. **Reclaim safely** — give the exact commands for the chosen approach, warning where truncating a non-append file or a database handle would corrupt data, and how to confirm space returned (`df -h` after). 5. **Prevent recurrence** — recommend the root-cause fix: a proper logrotate `copytruncate`/`postrotate` reopen, fixing an app that never closes files, or a monitoring check that alerts on growing deleted-file space. Output as: a confirmation of the diagnosis, a table of holders (PID, service, pinned size, safe reclaim action, exact command), and a prevention recommendation. Default to caution: never blindly truncate a file handle that is not a pure append log, prefer a graceful reopen/HUP over killing a process, and verify the process tolerates the action before reclaiming on a production database or stateful service.