🚨 Linux Error Guide: 'I/O error' on Write — Diagnose a Failing Disk and Bad Sectors

Overview

A write that fails at the block layer is one of the loudest warnings Linux will ever give you. When the kernel cannot commit a sector to the underlying device, it logs a blk_update_request (or, on older kernels, end_request) error and returns EIO to userspace. Applications and shells surface this as write error: Input/output error.

[  842.113905] blk_update_request: I/O error, dev sda, sector 1953120 op 0x1:(WRITE) flags 0x800 phys_seg 4 prio class 0
[  842.113942] Buffer I/O error on device sda1, logical block 244108, lost async page write
[  842.114001] EXT4-fs warning (device sda1): ext4_end_bio:344: I/O error 10 writing to inode 131076
[  842.221774] EXT4-fs error (device sda1): ext4_journal_check_start:83: Detected aborted journal
[  842.221802] EXT4-fs (sda1): Remounting filesystem read-only

Unlike a transient network or NFS hiccup, a persistent write I/O error almost always means the physical media, its controller, or the cable between them is failing. Treat it as a data-loss emergency, not a puzzle to retry your way out of.

Symptoms

• Kernel log lines containing blk_update_request: I/O error or Buffer I/O error with op 0x1:(WRITE).
• Userspace commands failing with write error: Input/output error or cannot write ... : Input/output error.
• The filesystem suddenly remounting read-only (Remounting filesystem read-only, Detected aborted journal), causing writes across the whole system to fail.
• Databases, log daemons, and package managers crashing or refusing to start because they cannot flush to disk.
• Growing latency and audible clicking on spinning disks, or a solid-state drive that has flipped to a read-only “end of life” state.
• smartctl reporting a failing self-assessment or climbing pending/reallocated sector counts.

Common Root Causes

• Bad sectors on a spinning disk. The drive tried to write a physical sector and failed. If it has spare sectors, it may transparently reallocate; once it runs out or the write path itself is degrading, it returns an error.
• Pending / reallocated sectors accumulating. Current_Pending_Sector counts sectors the drive is unsure about; Reallocated_Sector_Ct counts sectors already remapped. Steady growth means active media failure.
• A dying SSD. NAND wear-out, exhausted spare blocks, or controller failure. Many SSDs deliberately go read-only when their reserve is spent, turning every write into an EIO.
• Controller, cable, or backplane faults. A marginal SATA/SAS cable, a flaky HBA port, or a bad backplane slot produces I/O errors that look identical to media failure. dmesg link-reset messages (ata1: SATA link ..., hard resetting link) are the tell.
• Power / thermal issues. Undervoltage or overheating causes intermittent write failures that come and go with load.
• Filesystem-level fallout. Once writes fail, journaled filesystems (ext4, XFS) abort the journal and remount read-only to protect metadata — the read-only mount is a symptom, not the disease.

Diagnostic Workflow

Work from the loud symptom down to the physical device. Do not run repair or write tests on a disk you have not backed up. Your first job is to preserve data, then diagnose.

Confirm the errors and identify the device:

# Human-readable timestamps; find the failing device and sector
dmesg -T | grep -i 'i/o error'

# Kernel ring buffer via journald, including ATA link-reset noise
journalctl -k | grep -iE 'i/o error|ata[0-9]|hard resetting|remounting'

# Map the device name to partitions, mountpoints, and model
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,MODEL,SERIAL

Interrogate the drive’s own health with SMART:

# Full SMART report: overall health + the attribute table
smartctl -a /dev/sda

# The three attributes that matter most for write failures:
smartctl -A /dev/sda | grep -iE 'Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable'

Interpretation:

• Reallocated_Sector_Ct > 0 and rising — the drive is remapping bad sectors. A few may be tolerable; steady growth is terminal.
• Current_Pending_Sector > 0 — sectors that failed a read/write and await reallocation. This is the strongest correlate of active data loss.
• Offline_Uncorrectable > 0 — sectors the drive could not recover at all.
• A SMART overall-health self-assessment test result: FAILED! line means: replace now.

Run the drive’s built-in self-test (non-destructive, safe on mounted disks):

# Kick off the short self-test, then poll for the result
smartctl -t short /dev/sda
sleep 120
smartctl -l selftest /dev/sda

If SMART is clean but you still see errors, suspect cabling or the controller — reseat or swap the SATA/SAS cable and try a different port before condemning the disk.

Optionally, scan for bad blocks in read-only, non-destructive mode. Never use the destructive write test on a disk holding data:

# -s show progress, -v verbose. This is a READ scan — safe on live data.
badblocks -sv /dev/sda

# WARNING: adding -w runs a DESTRUCTIVE write test that WIPES the disk.
# Only ever use -w on a blank drive you intend to erase:
# badblocks -wsv /dev/sdX   # <-- DESTROYS ALL DATA. Do not run on live disks.

Example Root Cause Analysis

A logging host began throwing write error: Input/output error from its ingest daemon, and /var/log had gone read-only. dmesg -T | grep -i 'i/o error' showed repeated blk_update_request: I/O error, dev sda, sector ... op 0x1:(WRITE) lines, followed by EXT4-fs (sda1): Remounting filesystem read-only.

smartctl -a /dev/sda returned SMART overall-health self-assessment test result: PASSED, which is misleading — the overall flag lags behind attribute decay. Filtering the attribute table told the real story:

  5 Reallocated_Sector_Ct   0x0033   094   094   010    Pre-fail  Always  -  312
197 Current_Pending_Sector  0x0012   098   098   000    Old_age   Always  -  48
198 Offline_Uncorrectable   0x0010   098   098   000    Old_age   Offline -  48

312 already-reallocated sectors plus 48 pending and 48 uncorrectable, compared against a baseline from a week earlier showing single digits, confirmed active media failure. There were no ata1: hard resetting link messages, which ruled out the cable and pointed squarely at the media.

Action taken:

1. Stopped writing to the disk immediately — every write attempt risks touching more bad sectors and racing the drive to death.
2. Remounted read-only was already in effect; the data was evacuated by copying off the still-readable files with rsync -a --ignore-errors to a healthy target, prioritizing irreplaceable data first.
3. Because the host was part of a RAID1 mirror, the failing member was marked faulty and removed (mdadm --manage /dev/md0 --fail /dev/sda1 --remove /dev/sda1), the drive physically replaced, and the array rebuilt onto the new disk.
4. For hosts without RAID, the fix is the same shape: replace the disk and restore from the most recent backup.

The failing drive was retired, not reused. A disk with hundreds of reallocated sectors and rising pending counts will keep failing.

Prevention Best Practices

• Run smartd. Install smartmontools, enable the smartd service, and configure /etc/smartd.conf to run periodic self-tests and email on attribute changes — for example /dev/sda -a -o on -S on -s (S/../.././02|L/../../6/03) -m ops@example.com. Catching a rising Current_Pending_Sector a week early is the whole game.
• Alert specifically on reallocated and pending sectors. Don’t rely on the overall PASSED/FAILED flag; it trails reality. Scrape Reallocated_Sector_Ct, Current_Pending_Sector, and Offline_Uncorrectable into your metrics pipeline and alert on any nonzero delta.
• Use RAID for availability, not as a backup. A mirror or RAID6 array survives a single-disk write failure without downtime, but it does not protect against accidental deletion, corruption, or a two-disk failure. Scrub arrays regularly (echo check > /sys/block/md0/md/sync_action) to surface latent bad sectors before they bite during a rebuild.
• Keep tested, off-host backups. The only reliable recovery from a dead disk is a restore. Verify restores periodically — an untested backup is a hypothesis.
• Watch SSD wear. Monitor the SSD’s Percentage Used / media-wearout indicator and Available Spare, and budget replacements before drives approach their endurance limit.
• Track temperatures and check cabling on new hardware. A surprising share of “disk failures” are marginal cables or hot enclosures. Reseat and monitor before ordering replacement media.

Quick Command Reference

# Find write I/O errors and the affected device/sector
dmesg -T | grep -i 'i/o error'
journalctl -k | grep -iE 'i/o error|ata[0-9]|remounting'

# Map devices to mountpoints and models
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,MODEL,SERIAL

# SMART health and the attributes that matter
smartctl -a /dev/sda
smartctl -A /dev/sda | grep -iE 'Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable'
smartctl -H /dev/sda

# Run and read a non-destructive self-test
smartctl -t short /dev/sda
smartctl -l selftest /dev/sda

# Non-destructive read scan for bad blocks (NEVER add -w on live data)
badblocks -sv /dev/sda

# Evacuate data from a dying disk, skipping unreadable sectors
rsync -a --ignore-errors /mnt/failing/ /mnt/healthy/

# RAID: fail and remove a bad member, then rebuild after replacement
mdadm --manage /dev/md0 --fail /dev/sda1 --remove /dev/sda1

Conclusion

A write-path I/O error is the kernel telling you the media underneath a filesystem can no longer be trusted. The reflex to retry, fsck, or reformat is exactly wrong: those actions write to a disk that is already failing and can turn a partial loss into a total one. Instead, confirm the failing device with dmesg and lsblk, read the SMART attributes — especially Current_Pending_Sector and Reallocated_Sector_Ct — and rule out cabling with the ATA link messages. If the media is failing, stop writing, evacuate what you can read, and replace the drive, restoring from RAID or backup. The engineering that pays off is upstream: smartd monitoring, sector-level alerting, redundancy, and tested backups so the next dying disk is a scheduled swap instead of a 2 a.m. outage.