🚨 Kafka Error Guide: 'KafkaStorageException: Disk error when trying to access log file' — Recover the Offline Log Directory

Overview

A broker raises KafkaStorageException when an I/O operation against a log directory fails and Kafka marks that directory (and every partition on it) offline. It appears in the broker log during appends, flushes, or startup:

org.apache.kafka.common.errors.KafkaStorageException: Error while writing to checkpoint file /var/lib/kafka/data/replication-offset-checkpoint
[2026-07-09 10:14:02,551] ERROR Uncaught exception in scheduled task 'flush-log' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.
[2026-07-09 10:14:02,552] ERROR Error while writing to checkpoint file (kafka.server.LogDirFailureChannel)

Once a log directory fails, the broker stops serving the replicas stored there, and clients see KafkaStorageException or NotLeaderOrFollowerException for the affected partitions.

Symptoms

• Producers and consumers get KafkaStorageException or leadership errors for a subset of partitions while others work.
• UnderReplicatedPartitions and OfflineReplicaCount climb on the affected broker.
• The broker stays up but has marked one or more log.dirs entries offline (JBOD setups).
• Broker log shows Stopping serving replicas in dir <path> and LogDirFailureChannel messages.
• kafka-log-dirs.sh reports the directory with an error or missing from the online set.

Common Root Causes

• Physical disk failure or read-only remount — the underlying device threw an I/O error and the filesystem remounted read-only.
• Disk full — the log directory ran out of space or inodes, so writes and checkpoint updates fail.
• Permission change — the Kafka user lost write access to the log directory after an OS/ownership change.
• Corrupted or unwritable checkpoint files — replication-offset-checkpoint or recovery-point-offset-checkpoint cannot be written.
• Volume detached / mount lost — a network or cloud volume backing log.dirs disappeared while the broker ran.
• Filesystem errors — ext4/xfs metadata corruption surfacing as EIO on access.

Diagnostic Workflow

Identify which log directory the broker marked offline:

kafka-log-dirs.sh --bootstrap-server localhost:9092 \
  --describe --broker-list 3 | python3 -m json.tool | grep -i error

Check disk space and inodes on every log.dirs path:

df -h /var/lib/kafka/data
df -i /var/lib/kafka/data

Confirm the mount is healthy and writable, and check for read-only remounts:

mount | grep kafka
touch /var/lib/kafka/data/.write-test && rm /var/lib/kafka/data/.write-test
dmesg | grep -iE 'i/o error|ext4-fs error|xfs|remount'

Verify ownership and permissions match the Kafka service user:

ls -ld /var/lib/kafka/data
sudo -u kafka test -w /var/lib/kafka/data && echo writable || echo NOT-writable

Scope the impact and find under-replicated partitions caused by the offline dir:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions

Example Root Cause Analysis

A broker in a three-node cluster began throwing KafkaStorageException: Disk error when trying to access log file. Producers to a dozen partitions started failing while the rest of the cluster was fine, and kafka-log-dirs.sh showed /var/lib/kafka/data-1 with an error.

dmesg revealed EXT4-fs error entries and a read-only remount on the device backing data-1 — a failing cloud volume. Because those partitions had replication factor 3 with healthy replicas on the other two brokers, leadership had already moved and no data was lost, but the broker kept the directory offline.

The fix was to stop the broker, detach the faulty volume, attach a healthy one at the same mount path with correct ownership, and restart. The broker recreated the log directory, refetched the replicas from leaders, and UnderReplicatedPartitions returned to zero. Retention and RF ≥ 3 with min.insync.replicas=2 meant a single disk failure was survivable without data loss.

Prevention Best Practices

• Run replication factor ≥ 3 with min.insync.replicas=2 so a single failed log directory never causes data loss or unavailability.
• Alert on OfflineReplicaCount > 0 and UnderReplicatedPartitions > 0 so an offline log dir pages immediately.
• Monitor disk byte and inode usage per log.dirs path and alert before 85%; ENOSPC is a frequent cause.
• Watch dmesg/SMART for disk errors so you replace failing volumes before they take a directory offline.
• Pin Kafka data-dir ownership and permissions in configuration management so an OS change never revokes write access.
• Use JBOD with multiple log.dirs so one bad disk removes only its partitions, not the whole broker.

Quick Command Reference

# Which log dir is offline / errored
kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe --broker-list 3

# Space and inode check
df -h /var/lib/kafka/data && df -i /var/lib/kafka/data

# Kernel/filesystem disk errors
dmesg | grep -iE 'i/o error|ext4-fs error|remount'

# Writability as the kafka user
sudo -u kafka test -w /var/lib/kafka/data && echo writable || echo NOT-writable

# Impact scope
kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions

Conclusion

KafkaStorageException means a log directory failed an I/O operation and Kafka took it offline to protect data integrity. The cause is almost always physical: a failing disk, a full or read-only filesystem, a lost mount, or wrong permissions. Diagnose with kafka-log-dirs.sh, df, dmesg, and a write test, then repair or replace the storage and let replication refill the partitions. With RF ≥ 3 and min.insync.replicas=2, a single directory failure is a recoverable event rather than an outage.