🚨 Kafka Error Guide: 'Error while creating log directories' Log Dir Failure

Exact Error Message

When Kafka cannot initialize one of its data directories, it logs a directory-creation failure and, if no usable log dir remains, aborts startup:

[2026-06-29 10:02:51,774] ERROR Error while creating log directories for the following: /var/lib/kafka/data-2 (kafka.log.LogManager)
java.io.IOException: Failed to create data directory /var/lib/kafka/data-2
        at kafka.log.LogManager.$anonfun$createAndValidateLogDirs$1(LogManager.scala:131)
        at kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:129)
        at kafka.log.LogManager.<init>(LogManager.scala:91)
[2026-06-29 10:02:51,779] ERROR Disk error while writing to recovery point file in directory /var/lib/kafka/data-2 (kafka.server.LogDirFailureChannel)
[2026-06-29 10:02:51,901] FATAL Shutdown broker because none of the specified log dirs from /var/lib/kafka/data-2 can be created or validated (kafka.log.LogManager)

The underlying IOException often reads No such file or directory (the parent path does not exist) or Permission denied (the Kafka user cannot write there). On a full filesystem you may instead see No space left on device.

What the Error Means

At startup, LogManager walks every path listed in log.dirs (or the legacy single log.dir) and tries to create and validate each one. For each directory it checks that the path exists or can be created, that it is writable, and that it can acquire a .lock file. If a directory fails these checks, Kafka marks it offline. When all configured log dirs are unusable, the broker logs the FATAL Shutdown broker because none of the specified log dirs ... can be created or validated line and exits. If only some dirs fail (in a JBOD multi-disk setup), the broker may start with reduced capacity and the affected partitions go offline.

Common Causes

• Missing parent path: The directory or its parent does not exist and Kafka cannot create it (often No such file or directory), typically after a host rebuild or an unmounted volume.
• Wrong ownership or permissions: The path is not owned by the Kafka service user, so creation or writing fails with Permission denied. SELinux/AppArmor contexts can cause the same symptom.
• Unmounted or remounted volume: The mount that should back log.dirs is not mounted, so Kafka writes into (or is blocked by) the bare mount point on the root filesystem.
• Full disk: The filesystem is at 100%, so creating the recovery-point or .lock file fails with No space left on device.
• Stale lock file: A previous crashed broker left a .lock file held, or the dir is owned by another running broker instance.
• Read-only filesystem: The volume remounted read-only after an I/O error, making all writes fail.

How to Reproduce the Error

Point log.dirs at a directory the Kafka user cannot create or write, then start the broker:

# server.properties
log.dirs=/var/lib/kafka/data-1,/var/lib/kafka/data-2

If /var/lib/kafka/data-2 is missing and its parent is owned by root with no write permission for the Kafka user, startup reproduces Error while creating log directories ... Permission denied. Filling the backing filesystem to 100% reproduces the No space left on device variant.

Diagnostic Commands

Read the directory-creation failure and its cause from the broker log:

grep -nE "Error while creating log directories|none of the specified log dirs|No space left|Permission denied|No such file" /var/log/kafka/server.log | tail -30

Capture startup output from the service manager:

journalctl -u kafka --no-pager -n 150 | grep -iE "log dir|ioexception|permission|space"

See which log dirs the broker considers configured and their state (run against a healthy node or the failing broker once partially up):

kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe

Confirm the configured paths:

grep -E "^log.dirs?=" /etc/kafka/server.properties

Check existence, ownership, and permissions of each path (read-only):

ls -ld /var/lib/kafka/data-1 /var/lib/kafka/data-2

Check free space and whether the path is actually a separate mount:

df -h /var/lib/kafka/data-1 /var/lib/kafka/data-2 && findmnt /var/lib/kafka/data-2

Step-by-Step Resolution

1. Identify the exact failing path and the precise I/O error. The grep above tells you both the directory and whether it is missing, permission-denied, or full.
2. For No such file or directory: Confirm the parent exists and the intended volume is mounted (findmnt). If a volume is unmounted, mount it before starting Kafka — never let Kafka write into a bare mount point.
3. For Permission denied: Make the path (and parents up to the data root) owned by the Kafka service user with appropriate write permissions, and verify no SELinux/AppArmor denial is logged.
4. For No space left on device: Free space or expand the volume. Investigate retention settings if a topic grew unexpectedly; do not delete segment files by hand.
5. For a stale lock: Ensure no other Kafka JVM is running against the same dir, then remove the orphaned .lock only after confirming the previous process is gone.
6. For read-only filesystem: Check the kernel log for the I/O error that triggered the remount, repair or replace the disk, and remount read-write.
7. Restart and confirm the broker reaches started and all expected log dirs are online via kafka-log-dirs.sh --describe.

Prevention and Best Practices

• Pre-create every path in log.dirs with correct ownership as part of provisioning, before the broker ever starts.
• Use systemd mount dependencies (or equivalent) so Kafka cannot start before its data volumes are mounted.
• Monitor free space and inode usage on each log dir and alert at a threshold (e.g., 80%) so you act before a full disk forces the broker offline.
• In JBOD setups, monitor per-directory online/offline state so a single failed disk does not silently shrink capacity unnoticed.
• Keep retention (retention.ms, retention.bytes) sized to your actual disk so normal growth never fills the volume.
• For fast triage when a broker won’t start on a log-dir error, the free incident assistant can map the I/O error to a likely fix.

Related Errors

• Fatal error during KafkaServer startup — the broad startup-abort banner this log-dir failure ultimately triggers.
• KafkaStorageException / “Disk error while writing” — runtime I/O failures that take a log dir offline after startup.
• No space left on device — the specific full-disk variant of this error.
• InconsistentClusterIdException — a different meta.properties-related startup failure within the same data dir.

Frequently Asked Questions

Does the broker always die on this error? Only when all configured log dirs fail. In a multi-disk (JBOD) setup, if at least one dir is usable the broker can start with the failed dir marked offline and the partitions on it unavailable until you repair the disk.

Why does Kafka write into my root filesystem when the volume is unmounted? If the mount point directory exists but the volume is not mounted, the path resolves to the underlying root filesystem. Kafka happily writes there, fills root, and the real volume stays empty. Always gate Kafka startup on the mount being present.

Can I just chmod 777 the directory to fix Permission denied? Avoid it. Set correct ownership to the Kafka service user with normal directory permissions instead. World-writable data directories are a security risk and mask the real ownership problem.

Is it safe to delete the .lock file? Only after confirming no Kafka process is using that directory. The lock prevents two brokers from corrupting the same data dir. If the owning process is truly dead, removing the stale lock is safe; if it is alive, removal risks corruption.

How do I tell a full disk from a permissions problem? The error text is decisive: No space left on device means full (confirm with df -h), while Permission denied means ownership/permissions (confirm with ls -ld).