lsof Deleted-File Handle Leak Investigation Prompt
Diagnose a disk that shows free space in du but full in df by hunting down processes holding open handles to deleted files, then reclaim the space safely without killing critical services.
- Target user
- Linux sysadmins troubleshooting phantom disk-full conditions on servers
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior Linux engineer who immediately reaches for `lsof +L1` when `df` and `du` disagree, because the space is held by a process clinging to an unlinked file, not by anything on disk you can delete. I will provide: - `df -h` and `df -i` output and the filesystem that is reporting full - `du -sh` totals for the suspect mount (showing far less than df) - `lsof +L1` or `lsof -nP +L1 <mount>` output if available - Which services write to that filesystem (log files, app temp, databases) - Whether services on the box can be safely restarted Your job: 1. **Confirm the symptom** — verify this is a deleted-but-open-file situation (df full, du low, inodes fine) versus a genuine disk-full or inode-exhaustion problem, and rule out the alternatives. 2. **Identify the holders** — from `lsof +L1`, list the PIDs holding deleted files, the size each handle pins, the owning process/service, and the likely cause (rotated log not reopened, crashed-but-still-running writer, tmp file). 3. **Rank reclaim options** — order remediations by safety: truncate the open handle via `/proc/<pid>/fd/<n>` for an append-only log, signal the process to reopen its logs (`logrotate` postrotate, `kill -HUP`), or restart the service as a last resort; state which is safe for each holder. 4. **Reclaim safely** — give the exact commands for the chosen approach, warning where truncating a non-append file or a database handle would corrupt data, and how to confirm space returned (`df -h` after). 5. **Prevent recurrence** — recommend the root-cause fix: a proper logrotate `copytruncate`/`postrotate` reopen, fixing an app that never closes files, or a monitoring check that alerts on growing deleted-file space. Output as: a confirmation of the diagnosis, a table of holders (PID, service, pinned size, safe reclaim action, exact command), and a prevention recommendation. Default to caution: never blindly truncate a file handle that is not a pure append log, prefer a graceful reopen/HUP over killing a process, and verify the process tolerates the action before reclaiming on a production database or stateful service.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
cloud-init Debugging & Troubleshooting Prompt
Diagnose why a cloud instance came up wrong — user-data that never ran, SSH keys or hostname not applied, a module that failed silently, or first-boot vs re-run confusion — by reading cloud-init's logs, stages, and datasource correctly.
-
Shared Library Loader (ld.so) & LD_PRELOAD Debug Prompt
Debug 'error while loading shared libraries', wrong-version symbol errors, and unexpected library resolution by reasoning about the dynamic linker's search order, ldconfig cache, RPATH/RUNPATH, and LD_PRELOAD/LD_LIBRARY_PATH overrides.
-
ss Socket State & TCP Backlog Triage Prompt
Read ss output to explain a connection problem — stuck SYN-RECV/CLOSE-WAIT/TIME-WAIT piles, full accept/SYN backlogs, or exhausted ephemeral ports — and pinpoint whether the app or the kernel is to blame.
-
sosreport Diagnostic Bundle Review Prompt
Systematically read a RHEL/Rocky sosreport bundle to find the root cause of a performance, boot, or service incident without manually grepping hundreds of collected files.
More Linux Admins prompts & error guides
Browse every Linux Admins prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.