🚨 Docker Error Guide: 'no space left on device' Docker Disk Exhaustion

Exact Error Message

Docker raises this whenever a write into its data root cannot find free space. It shows up during pulls, builds, container starts, and image commits:

write /var/lib/docker/overlay2/3f9a.../diff/usr/lib/x86_64-linux-gnu/libcrypto.so.3: no space left on device

The tar-extraction variant appears during docker pull or docker build while unpacking layers:

failed to register layer: Error processing tar file(exit status 1): write /usr/share/icons/...: no space left on device

And inode exhaustion produces the same error text even when df -h shows free bytes:

mkdir /var/lib/docker/tmp/...: no space left on device

What It Means

Docker stores images, container writable layers, volumes, and the build cache under its data root (default /var/lib/docker). no space left on device is the kernel’s ENOSPC error: a write failed because the filesystem backing that directory has no free blocks or no free inodes left. Either the partition holding /var/lib/docker is genuinely full, or it has run out of inodes (millions of tiny files in overlay2 layers) while still showing free bytes. Until space is reclaimed, every operation that writes — pulling an image, starting a container, extending a log file — fails.

Common Causes

• Accumulated images, containers, and volumes — months of docker pull, stopped containers, and dangling volumes that nothing prunes.
• Build cache growth — BuildKit caches every layer; on busy CI hosts this can be tens of gigabytes.
• Inode exhaustion — overlay2 creates huge numbers of small files; you can run out of inodes long before bytes.
• A small /var/lib/docker partition — Docker installed on a tiny root volume while data is mounted elsewhere.
• Dangling layers — <none>:<none> images left after rebuilds.
• Unbounded container logs — the default json-file driver writes logs forever; a chatty container can fill the disk by itself.
• overlay2 directory growth — large writable layers from containers writing data into their own filesystem instead of a volume.

How to Reproduce the Error

On a host with a nearly full /var/lib/docker, pulling a large image exhausts the remaining space:

df -h /var/lib/docker
docker pull tensorflow/tensorflow:latest-gpu

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   30G   29G  120M  100% /
...
failed to register layer: Error processing tar file(exit status 1): write ...: no space left on device

Diagnostic Commands

Start with Docker’s own accounting to see where the space went:

docker system df
docker system df -v          # per-image / per-volume / build-cache breakdown

Confirm at the filesystem level — both bytes and inodes:

df -h /var/lib/docker
df -i /var/lib/docker        # inode usage; IUse% at 100% with free bytes = inode exhaustion

Find which subdirectory dominates the data root:

sudo du -sh /var/lib/docker/* | sort -rh | head
sudo du -sh /var/lib/docker/containers/*/*-json.log 2>/dev/null | sort -rh | head

Check the daemon’s data root and log driver, and read daemon errors:

docker info | grep -i 'Docker Root Dir\|Storage Driver\|Logging Driver'
journalctl -u docker --since '1 hour ago' | grep -i 'no space\|enospc'

Step-by-Step Resolution

1. Reclaim safely first. Remove only unused, reclaimable data before anything aggressive:

docker container prune -f
docker image prune -f          # dangling <none> images only
docker builder prune -f        # build cache

2. Prune build cache explicitly. On CI hosts the cache is usually the biggest offender:

docker buildx prune -af

3. Use the full prune with caution. -a removes all unused images and --volumes deletes unused volumes — including data you may want. Confirm nothing important is in an unused volume first:

docker volume ls
docker system prune -a --volumes

4. Cap container logs. Set log rotation in /etc/docker/daemon.json so json-file logs can never fill the disk again:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

sudo systemctl restart docker     # applies to newly created containers

Truncate an existing runaway log without recreating the container:

sudo truncate -s 0 $(docker inspect --format='{{.LogPath}}' <container>)

5. Free inodes if that is the limit. When df -i shows 100% IUse, deleting unused images/cache reclaims inodes too; a full prune -a is the fastest fix.

6. Move the data root to a larger volume. Point data-root at a bigger disk:

{ "data-root": "/mnt/docker" }

sudo systemctl stop docker
sudo rsync -aP /var/lib/docker/ /mnt/docker/
sudo systemctl start docker

7. Grow the disk. If usage is legitimate, expand the underlying volume (cloud disk resize + growpart/resize2fs/xfs_growfs) so /var/lib/docker has headroom.

How to Prevent the Issue

• Set max-size/max-file log limits in daemon.json on every host so container logs can never grow unbounded.
• Schedule a periodic docker system prune -af (and docker buildx prune -af on builders) via cron or a systemd timer, with --volumes only where safe.
• Monitor /var/lib/docker byte and inode usage and alert before 85%; inode exhaustion is invisible to byte-only alerts.
• Give /var/lib/docker its own generously sized volume rather than sharing the root partition.
• Mount application data into named volumes or bind mounts, not the container’s writable layer, to keep overlay2 lean.

Related Docker Errors

• failed to mount overlay: invalid argument — a different storage-layer failure on the same data root; see the failed to mount overlay guide.
• failed to solve with frontend dockerfile.v0 — a build that fails mid-solve, sometimes because the cache fills the disk; see the failed to solve guide.
• More storage and runtime fixes in the Docker guides.