🚨 Docker Error Guide: 'overlay2 failed to remove root filesystem: device or resource busy' — Clear Leaked Mounts

Overview

This error appears when Docker tries to delete a container’s storage but the kernel refuses because something still has the container’s merged directory mounted or open:

Error response from daemon: driver "overlay2" failed to remove root filesystem for 5f2c9a1b...: unlinkat /var/lib/docker/overlay2/9d4e7f.../merged: device or resource busy

device or resource busy is the kernel’s EBUSY. Docker cannot unlinkat (remove) the merged directory because it is still an active mount point or a process has a file open inside it. The container itself is usually already stopped — what remains is a leaked or lazy overlay mount that was never cleaned up, holding the directory hostage so docker rm cannot finish.

Symptoms

• docker rm <container> or docker rm -f <container> fails with the message above.
• Containers pile up in Removal In Progress or Dead state in docker ps -a.
• /var/lib/docker/overlay2/<hash>/merged still appears in /proc/mounts even though no container is running against it.
• Disk usage in /var/lib/docker/overlay2 keeps growing because storage for “removed” containers is never actually reclaimed.
• Frequently follows an ungraceful daemon restart, an OOM-killed container, or a bind-mount of a host path that was itself a mount point.

Common Root Causes

• A leaked overlay mount — the container stopped but its merged mount was never torn down, so the directory stays busy.
• A process on the host holding a file open inside the merged tree (a backup agent, antivirus, du, or a shell cd-ed into the path).
• Nested/lazy mounts — the container bind-mounted a host directory that was itself a mount point (e.g. an NFS share or another container’s volume), and the inner mount leaked into the host namespace.
• Systemd mount propagation — a MountFlags=slave/shared-propagation interaction that copies container mounts into other namespaces where they linger.
• A daemon that crashed mid-removal, leaving the storage layer half torn down.

Diagnostic Workflow

First find every mount that references the busy merged directory. Grep /proc/mounts for the container’s overlay hash from the error:

grep '9d4e7f' /proc/mounts
grep overlay2 /proc/mounts | grep merged

Find which process, if any, is holding a file open under that path:

sudo lsof +D /var/lib/docker/overlay2/9d4e7f.../merged 2>/dev/null
sudo fuser -vm /var/lib/docker/overlay2/9d4e7f.../merged

Confirm the container’s actual overlay directory and state via inspect:

docker inspect <container> --format '{{.GraphDriver.Data.MergedDir}}'
docker inspect <container> --format '{{.State.Status}}'

Read the daemon log for the removal failure and any earlier mount leak:

journalctl -u docker --since '30 min ago' | grep -i 'overlay2\|busy\|remove root'

If a stray mount is present, unmount it (lazily if a plain umount is also busy), then retry the removal:

sudo umount /var/lib/docker/overlay2/9d4e7f.../merged
# if still busy, detach lazily so the kernel cleans up when refs drop:
sudo umount -l /var/lib/docker/overlay2/9d4e7f.../merged
docker rm -f <container>

When mounts are truly stuck, restarting the daemon reconciles overlay2 state and releases leaked mounts:

sudo systemctl restart docker
docker rm -f <container>

Example Root Cause Analysis

A monitoring host accumulated dozens of Dead containers, and /var/lib/docker grew by several GB per day despite a nightly docker system prune. Every prune logged overlay2 failed to remove root filesystem ... device or resource busy.

grep merged /proc/mounts showed the leaked overlay mounts were still present. fuser -vm on one merged path returned nothing — no process held it open — which pointed at a leaked mount rather than an open file. The containers all shared one trait in docker inspect: they bind-mounted /mnt/nfs-logs, which was itself an NFS mount point. With the default mount propagation, each container’s view of that NFS mount leaked back into the host mount namespace, and when the container died the inner mount kept merged busy.

The immediate cleanup was a lazy unmount of each stuck merged directory followed by docker rm -f, then sudo systemctl restart docker to reconcile overlay2. The durable fix was to change the bind mount to use a private/rslave propagation so the NFS mount no longer leaked:

services:
  collector:
    image: myorg/collector:2.1
    volumes:
      - type: bind
        source: /mnt/nfs-logs
        target: /logs
        bind:
          propagation: rslave

After the propagation change the leaked mounts stopped appearing and disk reclamation returned to normal. The root cause was mount propagation from an NFS-backed bind mount, not a process holding files open — which the empty fuser output identified early.

Prevention Best Practices

• Avoid bind-mounting host paths that are themselves mount points (NFS, other volumes) without setting an explicit propagation mode (rslave/private).
• Never leave a shell or tooling cd-ed into /var/lib/docker/overlay2/*/merged; keep backup/AV scanners from traversing the Docker data root.
• Stop containers gracefully and avoid kill -9 on the daemon; ungraceful shutdowns are a common source of leaked mounts.
• Alert on growth of /var/lib/docker/overlay2 and on Dead containers so leaks are caught before disk fills.
• Keep the daemon and kernel current — several overlay2 mount-leak fixes have shipped over time.

Quick Command Reference

# Find mounts holding the busy merged dir (use the hash from the error)
grep '<hash>' /proc/mounts
grep overlay2 /proc/mounts | grep merged

# Who is holding it?
sudo lsof +D /var/lib/docker/overlay2/<hash>/merged
sudo fuser -vm /var/lib/docker/overlay2/<hash>/merged

# The container's actual merged dir
docker inspect <container> --format '{{.GraphDriver.Data.MergedDir}}'

# Unmount (lazy if needed) then force-remove
sudo umount /var/lib/docker/overlay2/<hash>/merged
sudo umount -l /var/lib/docker/overlay2/<hash>/merged
docker rm -f <container>

# Reconcile overlay2 state
sudo systemctl restart docker

Conclusion

overlay2 failed to remove root filesystem ... device or resource busy is an EBUSY on the container’s merged directory: the container is gone but a mount or open file still holds its storage. Diagnose in order — grep /proc/mounts for the overlay hash, use fuser/lsof to separate a leaked mount from an open file, unmount (lazily if necessary), then docker rm -f. A daemon restart clears stubborn cases. The lasting fix is upstream of removal: control mount propagation on bind mounts, keep processes out of the Docker data root, and shut containers and the daemon down gracefully so mounts never leak in the first place.