🦭 Podman Error: 'no space left on device' in containers/storage

Exact Error Message

$ podman pull docker.io/library/postgres:16
Trying to pull docker.io/library/postgres:16...
Getting image source signatures
Copying blob sha256:9b1...
Error: writing blob: adding layer with blob "sha256:9b1...":
processing tar file(write /usr/lib/postgresql/16/bin/postgres:
no space left on device): exit status 1

At container run or build time it surfaces differently:

Error: mounting storage for container 4f2a...: creating overlay mount to
/home/app/.local/share/containers/storage/overlay/abc.../merged:
no space left on device

What It Means

Podman keeps every image layer, container writable layer, and volume under a single directory called the graphroot. For rootless Podman that is ~/.local/share/containers/storage; for rootful Podman it is /var/lib/containers/storage. When the filesystem backing the graphroot fills up, every write — pulling a blob, extracting a tar layer, creating an overlay upper dir — fails with ENOSPC, which Podman surfaces verbatim as no space left on device.

The trap is that the message rarely tells you which filesystem is full. On a rootless setup the graphroot usually lives on /home, not /, so a df -h / showing plenty of free space proves nothing. There are also two distinct ways to run out: exhausting blocks, and exhausting inodes. Container images are enormous collections of small files, so overlay storage exhausts inodes surprisingly often while df -h still reports free gigabytes. A third variant hits rootless users specifically: the small tmpfs mounted at /run/user/$(id -u), where Podman writes runtime state, can fill even when the graphroot is fine.

Common Causes

• The partition holding the graphroot (/home for rootless, /var for rootful) is genuinely full.
• Accumulated dangling images and stale build cache from repeated podman build runs.
• Stopped containers still holding their writable overlay upper directories, plus orphaned volumes.
• Inode exhaustion on the graphroot filesystem despite free blocks (df -i at 100%).
• A single container writing large files into its writable layer instead of a volume, growing the overlay upper dir without bound.
• For rootless: the tmpfs at /run/user/UID is too small (default is a fraction of RAM) and fills with runtime state.

Diagnostic Commands

Ask Podman where its storage actually lives, then check that filesystem — not /:

podman info --format '{{.Store.GraphRoot}} {{.Store.RunRoot}} {{.Store.GraphDriverName}}'
df -h "$(podman info --format '{{.Store.GraphRoot}}')"
df -i "$(podman info --format '{{.Store.GraphRoot}}')"

Get Podman’s own accounting of what is consuming space and how much is reclaimable:

podman system df
podman system df -v

The -v output breaks down per image, per container, and per volume, and the RECLAIMABLE column tells you what a prune would actually recover. To find the heaviest directories inside the graphroot directly:

du -sh "$(podman info --format '{{.Store.GraphRoot}}')"/* 2>/dev/null | sort -h | tail -10

Check the rootless runtime tmpfs, which is a separate and much smaller filesystem:

df -h /run/user/$(id -u)
findmnt /run/user/$(id -u)

List what is still holding layers open:

podman ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Size}}'
podman images --filter dangling=true
podman volume ls

Step-by-Step Resolution

1. Confirm the right filesystem is the full one. If df -h on the graphroot shows 100% but df -h / is fine, you have found the problem:

GR="$(podman info --format '{{.Store.GraphRoot}}')"
df -h "$GR"; df -i "$GR"

2. Reclaim the cheap wins first — dangling images and stopped containers:

podman image prune -a
podman container prune

3. If that is not enough, do a full prune. --volumes also removes unused volumes, so check podman volume ls first — this deletes data:

podman system prune -a --volumes
podman system df

If a volume refuses to go because a container still references it, see volume is in use for how to find and release the holder.

4. If df -i shows 100% inodes with free blocks, pruning still helps, but the durable fix is a filesystem with dynamic inode allocation (XFS) or one formatted with a higher inode density:

df -i "$GR"
# on a fresh device, XFS allocates inodes dynamically:
sudo mkfs.xfs -f /dev/sdb1

5. Relocate the graphroot to a larger volume. Edit ~/.config/containers/storage.conf (rootless) or /etc/containers/storage.conf (rootful):

[storage]
driver = "overlay"
graphroot = "/data/containers/storage"
runroot = "/run/user/1000/containers"

Then wipe the old store and let Podman recreate it in the new location. podman system reset destroys all images, containers, volumes, and pods:

podman system reset
podman info --format '{{.Store.GraphRoot}}'

6. If the rootless runtime tmpfs is the culprit, raise its size. The per-user tmpfs size is controlled by logind:

sudo sed -i 's/^#\?RuntimeDirectorySize=.*/RuntimeDirectorySize=25%/' /etc/systemd/logind.conf
sudo systemctl restart systemd-logind
df -h /run/user/$(id -u)

Note that restarting systemd-logind ends user sessions; schedule it accordingly.

Prevention

• Put the graphroot on its own volume sized for your image churn, not on the root filesystem.
• Run podman image prune -a and podman container prune on a schedule via a systemd --user timer.
• Mount application data as named volumes or bind mounts so container writable layers stay small.
• Monitor both df -h and df -i on the graphroot filesystem; alert on inodes separately from blocks.
• Prefer XFS with project quotas (pquota) on the storage volume so a single runaway container cannot fill the device.
• Use multi-stage builds and --squash sparingly but deliberately to keep pulled and built layers compact.

To enforce a hard ceiling per container on XFS, mount the graphroot filesystem with project quotas enabled and set a size in storage.conf:

sudo mount -o remount,pquota /data

[storage.options.overlay]
size = "10G"

Related Errors

• volume is in use — a prune could not remove a volume because a container still references it. See volume is in use.
• fuse-overlayfs: command not found — a storage driver problem, not a capacity problem. See fuse-overlayfs not found.
• error creating overlay mount ... invalid argument — driver or kernel mismatch on the graphroot filesystem rather than ENOSPC.
• unable to write to database: disk I/O error — the containers/storage SQLite database could not be written, often the same full filesystem in a different guise.

Frequently Asked Questions

df -h / shows 60% free — why is Podman out of space? Rootless Podman stores everything under ~/.local/share/containers/storage, which is usually on /home. Always run df against the path reported by podman info --format '{{.Store.GraphRoot}}'.

Does podman system prune -a delete my data? It removes all unused images, stopped containers, networks, and build cache. Adding --volumes also removes volumes not attached to a container, which is real data. Inventory with podman volume ls first.

Why did pruning free almost nothing? Either the space is held by running containers’ writable layers, or you are out of inodes rather than blocks. Check df -i and podman system df -v to see where the usage actually sits.

Can I move storage without losing images? podman system reset after changing graphroot is the clean path but destroys everything. To preserve images, podman save them first, or rsync the old graphroot to the new path while Podman is stopped, preserving ownership and xattrs. For more storage fixes see the Podman guides.