🚨 Docker Error Guide: 'no space left on device' with Free Bytes — Fix Inode Exhaustion

Overview

Docker prints the classic ENOSPC message during pulls, builds, or container starts — but df -h shows the filesystem has plenty of free bytes. The give-away is df -i, which shows 100% inode usage:

write /var/lib/docker/overlay2/8c1d.../diff/usr/share/doc/pkg/changelog: no space left on device

$ df -h /var/lib/docker
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  200G   90G  100G  48% /var/lib/docker

$ df -i /var/lib/docker
Filesystem       Inodes   IUsed IFree IUse% Mounted on
/dev/nvme0n1p2  6553600 6553600     0  100% /var/lib/docker

The error text is identical to a bytes-full disk, but the cause is different: the filesystem has run out of inodes (the metadata slots that track individual files), not out of blocks. Every file needs one inode, so a filesystem can exhaust inodes while gigabytes of block space sit unused.

Symptoms

• docker pull/docker build fails with no space left on device while df -h shows free space.
• df -i reports IUse% at (or very near) 100% on the /var/lib/docker filesystem.
• touch /var/lib/docker/test fails with No space left on device despite free bytes.
• The host runs many small-file-heavy images (base OS layers, node_modules, Python venvs) or a busy CI builder.
• Removing a single large file frees bytes but does not fix the error, because the constraint is file count, not size.

Common Root Causes

• Overlay2 small-file explosion — each image layer unpacks thousands of tiny files; dozens of images multiply into millions of inodes.
• Accumulated images and build cache — months of docker pull and BuildKit layers, each consuming inodes even when compressed on disk is small.
• CI churn — builders that pull, build, and discard images all day accrue dangling layers faster than anything prunes them.
• A filesystem formatted with too few inodes — the volume was created with a low bytes-per-inode ratio (large expected files), so it runs out of inodes early.
• node_modules / language package trees baked into images — notorious for enormous file counts relative to their byte size.
• Log or temp files — many small rotated files under the data root or in container writable layers.

Diagnostic Workflow

First, confirm it is inodes, not bytes — run both side by side:

df -h /var/lib/docker      # bytes: may show plenty free
df -i /var/lib/docker      # inodes: IUse% at 100% confirms inode exhaustion

If df -h shows free space but df -i shows 100%, you have inode exhaustion — a different problem from the bytes-full case. Next, find which directory holds the most files (not the most bytes):

sudo bash -c 'for d in /var/lib/docker/overlay2 /var/lib/docker/containers /var/lib/docker/image; do echo -n "$d: "; find "$d" -xdev | wc -l; done'

Ask Docker how much it can reclaim:

docker system df
docker system df -v          # per-image and build-cache detail

Confirm the daemon’s data root and read its errors:

docker info | grep -i 'Docker Root Dir\|Storage Driver'
journalctl -u docker --since '30 min ago' | grep -i 'no space\|enospc'

If you plan to reformat, check the current inode ratio so you can choose a denser one:

sudo tune2fs -l /dev/nvme0n1p2 | grep -i 'Inode count\|Block count\|Inode size'

Example Root Cause Analysis

A CI builder began failing every image build with no space left on device. Operators saw df -h reporting 48% used and assumed a transient glitch. Running df -i told the real story: IUse% was 100% with zero free inodes. find /var/lib/docker/overlay2 | wc -l returned over six million entries — hundreds of Node.js and Python images had each unpacked tens of thousands of tiny package files.

Reclaiming unused images and cache immediately freed inodes:

docker system prune -af          # removes unused images + build cache
docker builder prune -af
df -i /var/lib/docker            # IUse% dropped from 100% to ~35%

Because the host would keep churning, the durable fix was to give /var/lib/docker a filesystem with more inodes. The volume had been formatted with the default ratio; recreating it with a smaller bytes-per-inode value roughly tripled the inode count:

sudo systemctl stop docker
sudo mkfs.ext4 -i 4096 /dev/nvme0n1p2   # one inode per 4 KiB = far more inodes
sudo mount /dev/nvme0n1p2 /var/lib/docker
sudo systemctl start docker

XFS is an alternative here — it allocates inodes dynamically and effectively never exhausts them under normal use.

Prevention Best Practices

• Alert on both df -h and df -i; a bytes-only monitor is blind to inode exhaustion.
• Schedule a periodic docker system prune -af and docker builder prune -af on builders so layer counts stay bounded.
• Slim images: use small base images, combine RUN steps, and remove package caches and node_modules build artifacts you do not ship.
• Format the /var/lib/docker volume with a higher inode density (mkfs.ext4 -i 4096) on small-file-heavy hosts, or use XFS for dynamic inode allocation.
• Keep application data in named volumes, not the container writable layer, so overlay2 stays lean.
• Track IUse% trend over time on CI builders and clean up before it reaches ~85%.

Quick Command Reference

# The key distinction: bytes vs inodes
df -h /var/lib/docker      # free bytes?
df -i /var/lib/docker      # 100% IUse% = inode exhaustion

# Which directory holds the most files
sudo find /var/lib/docker/overlay2 -xdev | wc -l

# Reclaim inodes by pruning unused images + cache
docker system prune -af
docker builder prune -af

# Inspect the current inode ratio (ext4)
sudo tune2fs -l /dev/<device> | grep -i 'Inode count'

# Reformat with more inodes (data volume, after stopping docker)
sudo mkfs.ext4 -i 4096 /dev/<device>

Conclusion

When Docker says no space left on device but df -h shows free bytes, do not chase disk space — run df -i. A 100% IUse% means inode exhaustion: the filesystem has run out of file-tracking slots, driven by overlay2’s mountain of tiny layer files. Prune unused images and build cache to reclaim inodes immediately, and for hosts that churn small files, reformat /var/lib/docker with a denser inode ratio or move to XFS. This is a fundamentally different failure from a bytes-full disk, and only df -i distinguishes them.

For the bytes-full counterpart and related storage issues, see the no space left on device (disk exhaustion) guide, the failed to register layer guide, and more fixes in the Docker guides.