🚨 Kafka Error Guide: 'Failed to acquire lock on file .lock in log directory' — Clear the Stale Broker Lock

Overview

Kafka creates a .lock file in each log directory so only one broker process ever owns that data. When a second process — or a restarting broker whose predecessor still holds the lock — tries to start, it fails immediately:

kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/kafka/data. A Kafka instance in another process or thread is using this directory.
[2026-07-09 07:12:55,004] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
[2026-07-09 07:12:55,006] INFO [KafkaServer id=1] shutting down (kafka.server.KafkaServer)

The broker refuses to start rather than risk two processes corrupting the same log. The fix is to find who holds the lock.

Symptoms

• A broker fails to start with Failed to acquire lock on file .lock and immediately shuts down.
• The failure repeats on every restart attempt until the lock holder is cleared.
• systemctl status kafka shows the service flapping between activating and failed.
• Two Kafka processes appear in ps, or a data directory is shared between brokers.
• After a hard crash, no Kafka process is running yet the lock still cannot be acquired.

Common Root Causes

• Duplicate broker process — an old Kafka JVM is still running and holds the lock while a new one tries to start.
• Two brokers sharing one log.dirs — misconfiguration points two broker instances at the same data directory.
• Fast restart before old process exited — the service restarted before the previous JVM fully released the lock.
• Container/orchestrator double-scheduling — two pods mount the same persistent volume for one broker id.
• Permission problem on the lock file — the Kafka user cannot read/replace an existing .lock owned by another user.
• Stale NFS lock — a network filesystem retains a lock after a crash (also a reason not to run log dirs on NFS).

Diagnostic Workflow

Check whether a Kafka process already holds the directory open:

ps -ef | grep -i '[k]afka.Kafka'
sudo fuser -v /var/lib/kafka/data/.lock

List every process with a handle on the log directory:

sudo lsof +D /var/lib/kafka/data 2>/dev/null | grep -i java

Confirm no second broker is configured against the same path:

grep -R 'log.dirs' /etc/kafka/server.properties

Check the lock file’s ownership and permissions against the Kafka service user:

ls -l /var/lib/kafka/data/.lock
sudo -u kafka test -w /var/lib/kafka/data/.lock && echo writable || echo NOT-writable

Read the startup failure context in the broker log:

grep -iE 'Failed to acquire lock|Fatal error during KafkaServer startup' /var/log/kafka/server.log | tail

Example Root Cause Analysis

A broker began crash-looping on startup with Failed to acquire lock on file .lock. systemctl kept restarting it, and each attempt failed the same way. On-call assumed disk corruption, but the data looked intact.

ps -ef | grep kafka revealed two Kafka JVMs for broker id 1: an original process that had hung during a previous shutdown (still holding the .lock) and the new one systemd kept launching. The hung JVM had stopped serving traffic but never exited, so it retained the file lock. Every restart spawned a new process that correctly refused to touch a directory another process owned.

The fix was to stop the systemd unit, confirm via fuser which PID held /var/lib/kafka/data/.lock, terminate that stale JVM, verify no process held the directory, then start the service cleanly. The broker acquired the lock and started normally. Nothing was corrupt — the lock did exactly its job of preventing two processes from writing the same logs. The follow-up was fixing the shutdown script so a broker fully exits before systemd restarts it.

Prevention Best Practices

• Ensure the service manager waits for the old JVM to fully exit before restarting (adequate TimeoutStopSec, proper shutdown handling).
• Never point two brokers at the same log.dirs; give every broker its own dedicated data directory and volume.
• In Kubernetes, use a StatefulSet so exactly one pod ever mounts a given broker’s persistent volume.
• Keep Kafka log directories on local/block storage, not NFS, to avoid stale network-filesystem locks.
• Pin lock-file and data-directory ownership to the Kafka user in configuration management.
• On a crash-loop, always check for a lingering Kafka process before assuming data corruption.

Quick Command Reference

# Is a Kafka process already running?
ps -ef | grep -i '[k]afka.Kafka'

# Who holds the lock file
sudo fuser -v /var/lib/kafka/data/.lock

# Processes with handles on the data dir
sudo lsof +D /var/lib/kafka/data | grep java

# Configured data directory
grep -R 'log.dirs' /etc/kafka/server.properties

# Lock file ownership
ls -l /var/lib/kafka/data/.lock

Conclusion

Failed to acquire lock on file .lock is a safety mechanism: Kafka will not let a second process write to a log directory another process already owns. The cause is almost always a lingering or duplicate broker JVM, two brokers sharing a data directory, or a fast restart racing the old process — not corruption. Find the lock holder with ps, fuser, and lsof, clear it, ensure one broker per data directory, and the broker starts cleanly.