🚨 Docker Error Guide: 'input/output error' on /var/lib/docker/overlay2 — Filesystem Corruption

Overview

Docker surfaces an input/output error when a read or write to a file under its overlay2 data root fails at the block-device level. It appears on pulls, builds, and container starts:

failed to register layer: open /var/lib/docker/overlay2/3f9a1c.../diff/usr/bin/app: input/output error

It also shows up when the daemon tries to list or mount a corrupted layer:

error while removing /var/lib/docker/overlay2/3f9a1c...: input/output error

input/output error is the kernel’s EIO: the filesystem asked the storage device for data and the device (or the filesystem structure) could not return it. Unlike no such file or directory, the path exists but cannot be read — this points at underlying disk or filesystem corruption, not a Docker logic error.

Symptoms

• Pulls, builds, or container starts fail with input/output error naming a path under /var/lib/docker/overlay2.
• docker image rm or docker system prune fails with EIO on specific layers.
• ls or du of certain overlay2 subdirectories hangs or errors.
• The host log (dmesg) shows I/O, EXT4/XFS, or block-device errors around the same time.
• The problem persists across daemon restarts because the data on disk is damaged.

Common Root Causes

• Failing or degraded disk — bad sectors on the volume backing /var/lib/docker.
• Filesystem corruption on the data-root partition after a crash, power loss, or unclean unmount.
• Underlying storage outage — a detached or flapping cloud/network block device.
• Overprovisioned thin LVM pool that ran out of space, corrupting the filesystem.
• A prior “no space left on device” event that left the filesystem structure inconsistent.
• RAID degradation where a member disk failed and parity could not reconstruct a block.

Diagnostic Workflow

First, confirm the errors are coming from the block layer, not Docker logic — read the kernel ring buffer:

sudo dmesg | grep -i error
sudo dmesg | grep -iE 'i/o error|ext4|xfs|blk_update_request|medium error'

Read the daemon log to tie the EIO to a specific layer and operation:

sudo journalctl -u docker --since '30 minutes ago' | grep -i 'input/output\|overlay'

Try to read the affected path directly; an EIO from a plain ls/cat confirms filesystem/disk damage rather than a Docker bug:

sudo ls -l /var/lib/docker/overlay2/3f9a1c.../diff 2>&1 | head

Check disk health and space, since full or failing disks are the usual cause:

df -h /var/lib/docker
sudo smartctl -H /dev/sda        # disk SMART health (needs smartmontools)

If the filesystem is corrupt, plan an offline check. Stop Docker first, then run the appropriate fsck for the filesystem type:

sudo systemctl stop docker
sudo umount /var/lib/docker        # only if it is a separate mount
sudo fsck -y /dev/sdb1             # ext4; use xfs_repair for XFS

Example Root Cause Analysis

A build host starts failing every pull with input/output error under /var/lib/docker/overlay2. Docker logic looks fine, so the kernel log is the first stop:

sudo dmesg | grep -iE 'i/o error|medium error'
# blk_update_request: I/O error, dev sdb, sector 194822144
# EXT4-fs error (device sdb1): ext4_find_entry: reading directory lblock

The block device sdb is throwing sector-level I/O errors and EXT4 has flagged directory corruption. This is hardware/filesystem damage, not a Docker problem. The recovery path depends on whether the disk is salvageable.

If SMART shows the disk is healthy and it was a one-off corruption (e.g. after a crash), an offline fsck can repair it:

sudo systemctl stop docker
sudo umount /var/lib/docker
sudo fsck -y /dev/sdb1
sudo mount /var/lib/docker && sudo systemctl start docker

If SMART reports the disk is failing, the correct action is to replace the disk, provision a fresh /var/lib/docker, and re-pull images — after restoring any named-volume data from backup. Because overlay2 layer data is disposable (it can be re-pulled), the priority is recovering volumes, not the layers themselves.

Prevention Best Practices

• Monitor disk SMART health and dmesg for I/O errors; alert before a marginal disk corrupts the data root.
• Keep byte and inode headroom on /var/lib/docker so a full-disk event never corrupts the filesystem.
• Ensure clean shutdowns and avoid hard power-offs that leave the filesystem inconsistent.
• Back up named volumes independently — overlay2 layers are re-pullable, but volume data is not.
• For thin-provisioned LVM, monitor pool utilization and never let the data pool fill.
• On cloud hosts, use durable block storage and watch for detach/attach flapping on the data-root volume.

Quick Command Reference

# Confirm block-level I/O errors from the kernel
sudo dmesg | grep -iE 'i/o error|ext4|xfs|medium error'

# Tie the EIO to a Docker layer/operation
sudo journalctl -u docker --since '30 minutes ago' | grep -i 'input/output'

# Check disk health and free space
sudo smartctl -H /dev/sdb
df -h /var/lib/docker

# Offline filesystem repair (Docker stopped first)
sudo systemctl stop docker
sudo fsck -y /dev/sdb1        # xfs_repair for XFS
sudo systemctl start docker

Conclusion

input/output error under /var/lib/docker/overlay2 is a storage-layer failure, not a Docker logic bug: the path exists but the disk or filesystem cannot read it. Confirm it with dmesg and a direct ls, check disk SMART health and free space, and either run an offline fsck/xfs_repair if the disk is sound or replace the disk and rebuild the data root if it is failing. Because overlay2 layers are re-pullable, focus recovery on named-volume backups. For more storage and runtime fixes, see the Docker guides.