🚨 Linux Error Guide: 'No space left on device' — Fix a Full Disk, Inodes, and Open Files

Overview

No space left on device is one of the most common — and most misdiagnosed — errors on a Linux host. The kernel returns the ENOSPC errno whenever a write cannot be completed, and userland tools surface it verbatim:

No space left on device

The naive assumption is “the disk is full, delete some files.” That is correct only about two-thirds of the time. There are three classic causes, and they demand different fixes:

1. The filesystem is out of bytes. The usual case — capacity is genuinely consumed.
2. The filesystem is out of inodes while bytes remain free. Millions of tiny files (mail spools, session files, cache shards) exhaust the inode table long before the block space runs out. df -h shows plenty of room; writes still fail.
3. Deleted-but-still-open files are pinned by a running process. A file was rm’d, but a process still holds an open file descriptor to it. The directory entry is gone, so du cannot see it, but the blocks are not released until the last fd closes. df and du disagree, and the disk stays full.

If you fix the wrong cause, the error comes right back. This guide walks the diagnostic path that distinguishes all three, plus the ext4 reserved-block gotcha that traps writes even when df shows a few percent free.

Symptoms

• Applications fail to write, log, or save with No space left on device or ENOSPC.
• touch newfile fails; even creating a zero-byte file errors out.
• Package managers (apt, dnf) abort mid-transaction.
• Databases go read-only or refuse new connections; PostgreSQL logs could not extend file, MySQL logs Error writing file.
• A service won’t start, or systemd reports it failed to write its state/journal.
• SSH sessions misbehave because shell history, temp files, or .Xauthority can’t be written.
• df -h sometimes shows the filesystem at 100% — and sometimes shows free space, which is the confusing case that points at inodes or open files.

Common Root Causes

• Runaway log files — an application or a stuck loop writing gigabytes to a single log under /var/log, often unrotated.
• /var filling up — package caches, container images, journald logs, mail queues, and database data all live here and grow silently.
• /tmp bloat — long-running processes leaving temp files that never get cleaned, especially where /tmp shares the root filesystem.
• journald with no cap — the systemd journal defaults to using up to ~10% of the filesystem, which is a lot on a large disk and can still balloon with verbose services.
• Inode exhaustion — a directory with millions of small files (session caches, .pyc files, spool directories) consumes every inode while leaving gigabytes of block space unusable.
• Deleted-but-open files — log rotation or a manual rm removed a file a daemon still has open; the space is only reclaimed when the fd is closed.
• ext4 reserved blocks — by default ext4 reserves 5% of the filesystem for root. Non-root writes fail at “95% full” even though df still shows a sliver free.
• Snapshots / thin provisioning — LVM snapshots or thin pools filling their metadata or data pool underneath a filesystem that looks fine from inside.

Diagnostic Workflow

Work top to bottom. Each step rules a cause in or out.

1. Confirm it’s actually a byte-full filesystem, and find which one:

df -h

Identify the mount that is at (or near) 100%. Remember which device backs the failing path — /, /var, and /home are frequently separate filesystems.

2. Rule out inode exhaustion:

df -i

If IUse% is 100% while df -h shows free space, you’re out of inodes, not bytes. The fix is to find and delete the directory full of tiny files (see step 4), not to free block space.

3. Find the biggest top-level consumers:

du -sh /* 2>/dev/null | sort -rh | head

Then drill into the offender. /var is the usual suspect — use -x to stay on one filesystem so you don’t wander into other mounts:

du -xh --max-depth=1 /var | sort -rh

Repeat --max-depth=1 down into /var/log, /var/lib, /var/cache until you land on the specific directory.

4. Locate deleted-but-still-open files (the “df and du disagree” case):

lsof +L1

+L1 lists open files whose link count has dropped below 1 — i.e. files that were unlinked (deleted) but are still held open. The SIZE/OFF column shows how much space each is pinning, and the PID/COMMAND columns tell you which process to restart or signal.

5. Hunt for individual large files:

find / -xdev -type f -size +500M

-xdev keeps find on a single filesystem so it doesn’t descend into /proc, /sys, or other mounts. Lower the threshold if nothing turns up.

6. Check the systemd journal footprint:

journalctl --disk-usage

If it’s large, cap and reclaim it (this deletes archived journal files immediately):

sudo journalctl --vacuum-size=200M
# or by age:
sudo journalctl --vacuum-time=7d

7. Check ext4 reserved blocks before you panic about the last few percent:

sudo tune2fs -l /dev/sdaX

Look at Reserved block count versus Block count. On ext4 the default is 5%, reserved for root and for keeping the filesystem defragmented. On a large data volume with no root-only services, that reservation is wasted headroom that non-root writers can’t touch.

Example Root Cause Analysis

A sanitized incident. An app server’s API started returning 500s; the logs showed No space left on device on every write.

Step 1 — capacity check. df -h was unambiguous:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   50G     0 100% /

Root filesystem at 100%, zero available. Classic full disk — or so it seemed.

Step 2 — where did it go? Summing the tree told a different story:

du -xsh / 2>/dev/null

du reported roughly 32G used, but df insisted 50G was consumed. An 18G discrepancy that du cannot see almost always means one thing: space held by deleted files that are still open.

Step 3 — confirm with lsof.

lsof +L1

The output pinned it:

COMMAND   PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK   NODE NAME
app-svc  4821  app    7w   REG  253,1  18.9G       0  131079 /var/log/app/output.log (deleted)

The application’s log had been rotated (the old file rm’d and a fresh one created), but the app-svc process was still writing to the old, deleted inode via file descriptor 7. Because the process never reopened its log after rotation, the kernel kept the 18.9G of blocks allocated. du couldn’t count it — there’s no directory entry — and the space would never come back on its own.

Step 4 — reclaim without a restart. A service restart closes the fd and frees the space instantly, and is the cleanest fix. But when a restart isn’t acceptable, you can truncate the file in place through the process’s own fd in /proc:

: > /proc/4821/fd/7

Redirecting empty output into /proc/PID/fd/7 truncates the still-open file to zero, releasing the blocks immediately while the process keeps its descriptor. df dropped straight back to ~64%. The permanent fix was configuring logrotate with copytruncate (or sending the daemon a reload signal in the postrotate hook) so rotation no longer strands an open file.

Prevention Best Practices

• Rotate every log with logrotate. Ship a config in /etc/logrotate.d/ for each noisy service, with rotate, maxsize, and compress. Critically, use copytruncate or a postrotate reload so daemons don’t keep writing to a deleted inode.
• Cap journald. Set SystemMaxUse= (e.g. SystemMaxUse=500M) in /etc/systemd/journald.conf so the journal can never eat an unbounded share of the disk, then systemctl restart systemd-journald.
• Monitor and alert below full. Page at 85% used, not 100% — you want runway to react. Free space at 100% is already an outage.
• Alert on inodes too. A byte-only monitor is blind to inode exhaustion. Track IUse% from df -i and alert on the same threshold.
• Put /var on its own partition. Isolating /var (or at least /var/log) means a log flood or runaway cache fills its own filesystem instead of taking down / and the whole host with it.
• Tune reserved blocks on data volumes. On large non-root filesystems, drop the 5% root reservation to 1% with tune2fs -m 1 /dev/sdaX to recover usable space (leave / alone).
• Clean caches proactively. Schedule package-cache cleanup (apt-get clean, dnf clean all) and use systemd-tmpfiles policies for /tmp.

Quick Command Reference

# Capacity and inodes
df -h                                      # byte usage per filesystem
df -i                                      # inode usage per filesystem

# Find what's using space
du -sh /* 2>/dev/null | sort -rh | head    # biggest top-level dirs
du -xh --max-depth=1 /var | sort -rh       # drill into /var (one fs)
find / -xdev -type f -size +500M           # large files on this fs

# Deleted-but-open files
lsof +L1                                   # unlinked files still held open
: > /proc/PID/fd/N                          # truncate one via its fd (no restart)

# journald
journalctl --disk-usage                    # current journal size
sudo journalctl --vacuum-size=200M         # shrink to a size cap
sudo journalctl --vacuum-time=7d           # drop entries older than 7d

# ext4 reserved blocks
sudo tune2fs -l /dev/sdaX                  # inspect reserved block count
sudo tune2fs -m 1 /dev/sdaX                # set root reservation to 1%

# Grow the filesystem after extending the block device
sudo resize2fs /dev/sdaX                    # ext2/3/4
sudo xfs_growfs /mount/point                # XFS (must be mounted)

Conclusion

No space left on device is really three problems wearing the same message. Before you delete anything, run df -h and df -i together: if inodes are maxed while bytes are free, you’re hunting a directory of tiny files, not big ones. If df reports full but du sums to noticeably less, the space is trapped in deleted-but-open files — find them with lsof +L1 and reclaim them by restarting the process or truncating through /proc/PID/fd/N. And when you’re staring at the last few percent, remember ext4’s 5% root reservation before you assume the disk is truly out.

The durable fix is prevention: rotate logs so daemons never strand an open inode, cap journald, isolate /var, and alert on both bytes and inodes at 85% so you’re growing the filesystem on your schedule — with resize2fs or xfs_growfs — instead of during an outage.