🚨 Linux Error Guide: 'Buffer I/O error on dev sda' — Recover From Read Errors and Bad Blocks

Overview

When a disk can no longer return the data stored in a sector, the Linux block layer surfaces it loudly. You’ll see paired kernel messages: one from the low-level driver reporting the failed request, and one from the buffer cache reporting that a logical block could not be read.

blk_update_request: I/O error, dev sda, sector 234881024 op 0x0:(READ) flags 0x0 phys_seg 8 prio class 0
Buffer I/O error on dev sda, logical block 29360128, async page read

The critical detail is the operation: op 0x0:(READ). This is a read failure — the drive tried to fetch data that already exists on the platter (or NAND) and physically could not. That is a very different situation from a write error, where the drive fails to store new data. This guide focuses exclusively on read errors: identifying unreadable sectors, rescuing whatever data is still readable, mapping bad sectors back to affected files, and forcing sector remapping. (A sibling post covers write errors, which usually point to a full spare-sector pool or a dying drive with no reserve capacity left.)

Read errors matter because they represent data you may lose permanently. The moment you see them, treat the disk as failing and prioritize imaging over in-place repair.

Symptoms

• Kernel log spam: Buffer I/O error on dev sda, logical block N and blk_update_request: I/O error ... op 0x0:(READ).
• Applications and shell commands returning Input/output error (EIO) when reading specific files: cat file.log fails, cp aborts partway, tar reports read errors on certain members.
• ls works but reading a file hangs for several seconds (the drive retries internally) before failing.
• dmesg shows SCSI/ATA messages like sd 0:0:0:0: [sda] tag#... UNC or Unrecovered read error - auto reallocate failed.
• SMART reports a non-zero and often growing Current_Pending_Sector count.
• The system may stay up if the affected sectors belong to data files, or crash/remount read-only if they belong to filesystem metadata.

Common Root Causes

• Media degradation / bad blocks: The most common cause. A sector’s magnetic (or NAND) integrity has decayed below the ECC’s ability to recover it. The drive knows the data is bad but has no clean copy to remap to, so it returns an uncorrectable read error (UNC).
• Uncorrectable ECC on an aging HDD: Sectors rot over time, especially in rarely-read cold data. The pending count grows as more sectors cross the threshold.
• Failing SSD NAND: Worn flash cells or a failing FTL produce the same block-layer errors, though SMART attributes differ (look at media wearout / reallocated NAND).
• Cable, connector, or controller faults: A flaky SATA cable or backplane can produce transient read errors with CRC error growth (UDMA_CRC_Error_Count) but no pending sectors — this is not media failure and is recoverable by reseating/replacing hardware.
• Filesystem corruption vs. media failure: If the disk reads fine (SMART clean, badblocks clean) but you still get EIO, the problem may be inconsistent filesystem metadata rather than bad media. Distinguishing these two is the first fork in the diagnostic path — never run fsck on a disk that is physically failing, as the writes it issues can accelerate the loss.

Diagnostic Workflow

Work from cheapest/safest to most invasive. Do not write to a suspect disk until you have an image.

1. Pull the kernel evidence with timestamps:

# Human-readable timestamps; catch both message styles
dmesg -T | grep -i 'buffer i/o\|i/o error'

# See the raw ATA/SCSI errors around the failure (UNC, sense keys)
dmesg -T | grep -iE 'sda|ata|unrecovered|medium error'

2. Read the drive’s own health data:

smartctl -a /dev/sda

Focus on these attributes:

• Current_Pending_Sector — sectors that failed a read and are queued for reallocation on next write. A non-zero value confirms unreadable media.
• Offline_Uncorrectable — sectors the drive could not recover during offline scanning.
• Reallocated_Sector_Ct — already-remapped sectors; rapid growth signals a dying drive.
• Raw_Read_Error_Rate / UDMA_CRC_Error_Count — high CRC errors with zero pending sectors point at the cable, not the platter.

3. Run a long self-test to let the drive scan its own media:

smartctl -t long /dev/sda
# Check estimated completion, then poll:
smartctl -l selftest /dev/sda

A completed long test reports the LBA of the first read failure, which you can map to a file later.

4. Confirm bad blocks in read-only, non-destructive mode:

# -s show progress, -v verbose. Default is a READ-ONLY test — safe.
badblocks -sv /dev/sda

# Optionally record the list for e2fsck to avoid those blocks later:
badblocks -sv -o /root/badblocks.txt /dev/sda

Never use badblocks -w (write mode) on a disk with data — it is destructive.

5. Pinpoint the failing offset with dd (read-only):

# Read the whole device but never stop; note where it errors in dmesg.
# conv=noerror keeps going, sync pads unreadable blocks with zeros.
dd if=/dev/sda of=/dev/null bs=4096 conv=noerror,sync status=progress

Watch dmesg for the reported sector N; that LBA is your bad block.

6. Image the dying disk with ddrescue (the important step):

dd is a blunt instrument for failing hardware. Use GNU ddrescue, which reads the good areas first, logs its progress to a mapfile, and only then retries the hard sectors — minimizing stress on a drive that may not survive many more reads.

# First pass: grab everything easy, skip errors fast (-n = no scraping).
# ALWAYS keep the mapfile; it lets you resume and refine.
ddrescue -f -n /dev/sda /mnt/rescue/sda.img /mnt/rescue/sda.map

# Second pass: retry the failed regions a few times with scraping.
ddrescue -f -r3 /dev/sda /mnt/rescue/sda.img /mnt/rescue/sda.map

Note -f is required when the destination is a device; here it’s a file so it forces overwrite. The mapfile is what makes ddrescue resumable and safe — never delete it mid-recovery.

7. Map a bad sector back to a filesystem and file (ext4):

Convert the device LBA (sector) reported by the kernel into a filesystem block, then into an inode and a path.

# Determine filesystem block size and partition start.
tune2fs -l /dev/sda1 | grep 'Block size'
fdisk -lu /dev/sda            # note the partition's start sector

# Compute the fs block: (bad_sector - partition_start) * 512 / blocksize
# Then ask debugfs which inode owns that block:
debugfs -R 'icheck <fs_block_number>' /dev/sda1

# Turn the inode number into a path:
debugfs -R 'ncheck <inode_number>' /dev/sda1

If icheck reports the block is free (no inode), the bad sector holds no live data — you can simply force a write to it to trigger remapping. If it maps to a file, you know exactly what you’re about to lose and can restore that file from backup.

8. Force remapping of a free/expendable sector:

Once you know a sector holds no needed data (or you’ve recovered the file), writing to it forces the drive to remap it from its spare pool, clearing the pending count.

# Overwrite exactly one 512-byte sector by LBA. DESTRUCTIVE to that sector only.
hdparm --write-sector <LBA> --yes-i-know-what-i-am-doing /dev/sda

# Or write the containing filesystem block via dd (know your offsets!):
dd if=/dev/zero of=/dev/sda1 bs=4096 seek=<fs_block> count=1 oflag=direct

After a successful write, re-check smartctl -a — Current_Pending_Sector should drop and Reallocated_Sector_Ct may tick up.

Example Root Cause Analysis

A logging host started throwing Input/output error when logrotate tried to compress yesterday’s app.log. The application stayed up; only that one file was unreadable.

Evidence: dmesg -T | grep -i 'i/o error' showed a repeating blk_update_request: I/O error, dev sda, sector 234881024 op 0x0:(READ) alongside Buffer I/O error on dev sda, logical block 29360128. smartctl -a /dev/sda reported Current_Pending_Sector = 8 and Reallocated_Sector_Ct = 0 — real media failure, not a cable (CRC error count was zero).

Scope check: A smartctl -t long completed with “read failure” at LBA 234881024, confirming a single bad region. Mapping it with debugfs -R 'icheck ...' then ncheck resolved to /var/log/app.log — exactly the file logrotate choked on.

Recovery decision: With pending sectors present, in-place repair was off the table. We attached a spare disk, ran ddrescue -f -n /dev/sda /mnt/rescue/sda.img /mnt/rescue/sda.map to capture 99.98% of the drive immediately, then ddrescue -f -r3 to retry the hard region. The bad sectors fell inside app.log; every other file imaged cleanly.

Resolution: We replaced the failing disk, restored the OS from the ddrescue image onto the new drive, and pulled the single corrupted app.log from the previous night’s backup. The old disk was retired rather than remapped — with pending sectors already appearing on cold data, its remaining life wasn’t worth the risk. Total data loss: a few hours of one log file.

Prevention Best Practices

• Run smartd: Enable smartd (from smartmontools) with DEVICESCAN and email alerts so Current_Pending_Sector growth pages you before a file becomes unreadable.
• Schedule long self-tests: In /etc/smartd.conf, add a directive like /dev/sda -a -s L/../../6/03 to run a long self-test weekly. Drives that scan themselves catch decaying sectors early, while a good copy may still exist to remap from.
• Scrub redundant storage: On mdraid, schedule echo check > /sys/block/mdX/md/sync_action (most distros ship a monthly cron/timer). On Btrfs run btrfs scrub start /mnt; on ZFS run zpool scrub tank. Scrubs read every block, and with redundancy present they repair bad sectors automatically instead of just reporting them.
• Keep tested backups: SMART and scrubbing reduce risk but never eliminate it. A read error on a single-disk system is only recoverable if you have a backup — automate them and test restores.
• Retire drives on pending-sector growth: A one-off remapped sector is normal wear; a steadily climbing pending or reallocated count means the drive is on its way out. Replace proactively.

Quick Command Reference

# Evidence
dmesg -T | grep -i 'buffer i/o\|i/o error'
dmesg -T | grep -iE 'sda|unrecovered|medium error'

# Health
smartctl -a /dev/sda                       # Current_Pending_Sector, Offline_Uncorrectable
smartctl -t long /dev/sda                   # start long self-test
smartctl -l selftest /dev/sda               # read self-test results

# Non-destructive scan
badblocks -sv /dev/sda                      # READ-ONLY bad block scan
dd if=/dev/sda of=/dev/null bs=4096 conv=noerror,sync status=progress

# Rescue (image first, repair never)
ddrescue -f -n /dev/sda /mnt/rescue/sda.img /mnt/rescue/sda.map
ddrescue -f -r3 /dev/sda /mnt/rescue/sda.img /mnt/rescue/sda.map

# Map bad sector -> file (ext4)
tune2fs -l /dev/sda1 | grep 'Block size'
debugfs -R 'icheck <fs_block>' /dev/sda1
debugfs -R 'ncheck <inode>' /dev/sda1

# Force remap of an expendable sector
hdparm --write-sector <LBA> --yes-i-know-what-i-am-doing /dev/sda

Conclusion

Buffer I/O error on dev sda, logical block N with op 0x0:(READ) is the kernel telling you a sector can no longer be read — data is at risk. The correct order of operations is: gather evidence (dmesg, smartctl), confirm whether it’s media (pending sectors) or hardware (CRC errors) or filesystem, and if the media is failing, image before you repair with ddrescue. Only after your data is safe should you map bad sectors to files, force remapping, or retire the drive. Treat the first read error as a warning shot: back up, replace the disk, and let smartd and scheduled scrubs catch the next one before it costs you a file.