🚨 Linux Error Guide: 'No space left on device' ENOSPC Disk and Inode Exhaustion

Overview

No space left on device is the ENOSPC errno (28). The kernel returns it whenever a write cannot be completed because the target filesystem has no room. The confusing part is that “room” means two separate resources: free data blocks (raw bytes) and free inodes (the metadata slots that track each file). A filesystem can show plenty of free gigabytes and still fail every write because it has run out of inodes — or be 100% full of blocks while inodes are abundant. Either condition surfaces with the identical message.

You will see it from almost any program that writes:

write error: No space left on device
cp: error writing '/var/log/app/out.log': No space left on device
-bash: cannot create temp file for here-document: No space left on device
OSError: [Errno 28] No space left on device

It occurs on any write path: creating or extending a file, tar/cp/rsync, package installs, database commits, and even shell here-documents. The failure is per-filesystem, so a full /var can break logging while /home has terabytes free.

Symptoms

• Writes, package installs, and log rotation fail with “No space left on device”.
• Services crash or refuse to start because they cannot create lock/pid/temp files.
• df -h may show 100% use on the affected mount — or it may not (inode or deleted-file cases).
• New files cannot be created even though df -h reports free space.

df -h /var
df -i /var

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        20G   20G     0 100% /var

Filesystem     Inodes  IUsed IFree IUse% Mounted on
/dev/sda3      1310720 487213 823507  38% /var

Here blocks are 100% full while inodes are fine — a classic byte-exhaustion case. Always check both df -h and df -i.

Common Root Causes

1. The filesystem is genuinely full (data blocks exhausted)

The simplest case: the mount holding the write target has no free bytes. Identify which filesystem is full and confirm Avail is 0 / Use% is 100%.

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   12G   36G  25% /
/dev/sda3        20G   20G     0 100% /var
/dev/sdb1       100G   18G   77G  19% /home

/var is full; anything writing under /var (logs, spool, databases) gets ENOSPC.

2. Inode exhaustion (free space but no inodes)

Each file/dir/symlink consumes one inode. Millions of tiny files (session caches, mail queues, PHP sessions) can exhaust inodes while leaving most of the disk empty. df -h looks healthy; df -i tells the truth.

df -i

Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/sda1      3276800  198114 3078686    7% /
/dev/sda3      1310720 1310720       0  100% /var

IUse% is 100% on /var while df -h would show free gigabytes. To find the offending directory:

sudo find /var -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -rn | head

 988231 /var/spool/postfix
  142880 /var/lib/php/sessions
   38104 /var/cache/nginx

3. A few very large files

One or two oversized files (an unrotated log, a core dump, a stray VM image) can fill the disk. Walk the tree by size from the mount point down.

sudo du -xh / --max-depth=1 2>/dev/null | sort -rh | head
sudo find /var -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

14G   /var/log
3.1G  /var/lib
512M  /var/cache
-rw-r--r-- 1 syslog adm 13G Jun 23 09:41 /var/log/syslog

A single 13G syslog is eating the partition.

4. Deleted-but-open files held by a process

A process can hold an open file descriptor to a file that was rm-ed. The directory entry is gone (so du cannot see it) but the blocks are not freed until the process closes the fd or exits. df shows full; du shows much less.

sudo lsof +L1
sudo lsof -nP | grep '(deleted)' | head

COMMAND   PID  USER   FD   TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
java     2417 tomcat  37w  REG  8,3   9663676416     0  131074 /var/log/app/catalina.out (deleted)

The deleted catalina.out still holds ~9G. Truncating the fd or restarting the process reclaims it:

sudo truncate -s 0 /proc/2417/fd/37

5. Reserved blocks for root (non-root writes fail first)

ext2/3/4 reserve a percentage of blocks (default 5%) for root. Once a filesystem hits that reserve, non-root processes get ENOSPC even though df shows a sliver of Avail usable only by root.

sudo tune2fs -l /dev/sda3 | grep -Ei 'Reserved block count|Block count'

Block count:              5242880
Reserved block count:     262144

262144 of 5242880 blocks (5%) are root-only. On a large data volume that is wasted space; lower it with tune2fs -m 1 /dev/sda3 (do not do this on /).

6. Runaway journald / application logs

systemd-journald or an app log can grow until it fills /var. Check journal disk usage and per-directory log size.

journalctl --disk-usage
sudo du -xh /var/log --max-depth=1 | sort -rh | head

Archived and active journals take up 8.0G in the file system.
8.1G  /var/log/journal
2.4G  /var/log/nginx

Vacuum the journal to reclaim space immediately:

sudo journalctl --vacuum-size=500M

Diagnostic Workflow

Step 1: Identify which filesystem is out of space

df -h
df -i

Compare both. Use% = 100% in df -h is block exhaustion; IUse% = 100% in df -i is inode exhaustion. Note the exact mount point.

Step 2: If blocks are full, find the largest consumers

sudo du -xh <MOUNT> --max-depth=1 2>/dev/null | sort -rh | head
sudo find <MOUNT> -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

-xdev keeps du/find on the one filesystem so you do not chase other mounts.

Step 3: Reconcile df vs du to catch deleted-open files

sudo lsof +L1

If df reports far more used than du accounts for, a process is holding deleted files open. Truncate the fd or restart the process.

Step 4: If inodes are exhausted, find the file-count hotspot

sudo find <MOUNT> -xdev -type f | cut -d/ -f1-5 | sort | uniq -c | sort -rn | head

This counts files per subtree to pinpoint the directory creating millions of small files.

Step 5: Reclaim and verify

sudo journalctl --vacuum-size=500M
sudo truncate -s 0 /var/log/<HUGE_LOG>   # keeps the open fd valid
df -h <MOUNT> && df -i <MOUNT>

Prefer truncate -s 0 over rm for a log a service still has open, so you do not create a deleted-but-held file.

Example Root Cause Analysis

A web server starts throwing OSError: [Errno 28] No space left on device and nginx 5xx errors. df -h /var shows 100% used, so it looks like a plain full disk. But du -xh /var --max-depth=1 only accounts for about 6G on a 20G partition — the numbers do not add up.

The mismatch between df (full) and du (only 6G) points at deleted-but-open files:

sudo lsof +L1

COMMAND  PID    USER   FD   TYPE DEVICE   SIZE/OFF NLINK  NODE NAME
python  3391 appuser  14w  REG  8,3   13958643712     0 262145 /var/log/app/debug.log (deleted)

A deploy script had rm-ed debug.log to “clear” it, but the running app still held the fd, so ~13G of blocks were never released. du could not see the file because its directory entry was gone, while df still counted the blocks.

Fix: truncate the live descriptor to release the blocks without restarting the app, then fix the rotation to truncate instead of delete:

sudo truncate -s 0 /proc/3391/fd/14
df -h /var

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        20G  6.2G   13G  34% /var

Space is reclaimed instantly and nginx recovers. The lasting fix is a logrotate copytruncate policy so log clearing never orphans an open fd again.

Prevention Best Practices

• Monitor both block and inode usage (df -h and df -i) and alert before 90% on either — inode exhaustion is the one teams forget.
• Configure logrotate with copytruncate (or signal the daemon to reopen) so rotating a log never leaves a deleted-but-open file holding the disk.
• Cap journald with SystemMaxUse= in journald.conf instead of relying on manual vacuuming.
• Put /var, /tmp, /home, and database data on separate filesystems so one runaway writer cannot take down the whole host.
• Clean ephemeral file factories (PHP sessions, mail spool, build caches) on a schedule before they exhaust inodes.
• Lower the ext4 reserved-block percentage on large non-root data volumes (tune2fs -m 1) to stop wasting 5% — never touch /.
• For broader disk and filesystem playbooks, see the Linux admins guides.

Quick Command Reference

# Block vs inode usage — always check both
df -h
df -i

# Largest directories / files on one filesystem
sudo du -xh <MOUNT> --max-depth=1 2>/dev/null | sort -rh | head
sudo find <MOUNT> -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

# Inode hotspot (millions of small files)
sudo find <MOUNT> -xdev -type f | cut -d/ -f1-5 | sort | uniq -c | sort -rn | head

# Deleted-but-open files holding space (df > du)
sudo lsof +L1
sudo lsof -nP | grep '(deleted)'
sudo truncate -s 0 /proc/<PID>/fd/<FD>

# Reserved-block check (ext4)
sudo tune2fs -l /dev/<DEV> | grep -Ei 'Reserved block count|Block count'

# journald reclaim
journalctl --disk-usage
sudo journalctl --vacuum-size=500M

# Clear a live log without orphaning the fd
sudo truncate -s 0 /var/log/<HUGE_LOG>

Conclusion

No space left on device (ENOSPC) means a write had nowhere to land — but “nowhere” can be empty data blocks or empty inodes, and the message is the same for both. Always check df -h and df -i first, then reconcile against du to catch deleted files a process still holds open. The usual root causes:

1. The filesystem is genuinely full of data blocks.
2. Inodes are exhausted by millions of small files while bytes remain free.
3. One or a few very large files (unrotated logs, core dumps, images).
4. Deleted-but-open files whose blocks a running process still holds.
5. ext4 reserved-blocks make non-root writes fail before df hits zero.
6. Runaway systemd-journald or application logs filling /var.

Identify the affected mount and which resource (blocks vs inodes) is exhausted, reclaim with truncate/vacuum, then fix rotation and monitoring so it does not recur.