🚨 Kafka Error Guide: 'Leader epoch mismatch' Fencing in the KRaft Metadata Log

Exact Error Message

After a controller election, KRaft fences stale participants by comparing leader epochs. A request carrying an out-of-date epoch is rejected with a mismatch, visible in controller.log:

[2026-06-29 12:30:41,556] WARN [RaftManager id=3] Leader epoch mismatch on Fetch from voter 3: request leaderEpoch 486 but current leaderEpoch is 488 (org.apache.kafka.raft.KafkaRaftClient)
[2026-06-29 12:30:41,557] INFO [RaftManager id=3] Rejecting fetch with FENCED_LEADER_EPOCH; follower must truncate to diverging offset 9,861,540 (org.apache.kafka.raft.KafkaRaftClient)
[2026-06-29 12:30:41,901] WARN [BrokerToControllerChannel id=21] Received FENCED_LEADER_EPOCH from controller; refreshing controller leadership (epoch 486 -> 488) (kafka.server.NodeToControllerRequestThread)

Brokers may also log:

org.apache.kafka.common.errors.FencedLeaderEpochException: Leader epoch 486 is older than the current epoch 488 for __cluster_metadata-0

What the Error Means

Every time a new controller is elected for the __cluster_metadata Raft group, the leader epoch increments. The epoch is a monotonically increasing fence: any node still acting on a previous epoch (a stale leader, or a follower that fetched from the old leader) is rejected with FENCED_LEADER_EPOCH. “Leader epoch mismatch” means a request arrived stamped with an epoch older than the quorum’s current one, so the receiver refuses it and tells the sender to refresh leadership and, if its log diverged, truncate to the agreed offset.

In a healthy cluster this is normal and self-healing: a brief flurry of mismatch warnings right after an election, then everything converges on the new epoch. It becomes a problem only when mismatches persist — indicating a node stuck on an old epoch, a split-brain attempt by a demoted leader, or repeated elections churning the epoch faster than nodes can catch up.

Common Causes

• Normal post-election fencing — a recent controller failover bumps the epoch; stale fetches are briefly rejected then refreshed. Transient and expected.
• A demoted/zombie leader that did not realize it lost leadership and keeps acting on the old epoch (network partition healing).
• Repeated elections (see leader-election failures) bumping the epoch so often that followers and brokers cannot settle on a current one.
• A follower with a divergent log tail that must truncate to the leader’s diverging offset before it can resume.
• Brokers caching a stale controller and sending requests with an old epoch after a quick failover.
• Clock skew or long GC pauses causing a node to lag epoch updates and repeatedly send stale requests.

How to Reproduce the Error

Force a controller failover and observe fencing of the old leader’s in-flight requests:

# Find the current leader
kafka-metadata-quorum.sh --bootstrap-controller c1:9093 describe --status

# Lab only: stop the active controller to trigger an election (epoch bump)
sudo systemctl stop kafka   # on the current leader node

# Watch the new leader fence stale-epoch requests, then converge
sudo journalctl -u kafka -f | grep -iE 'epoch|FENCED|truncate'

You will see a burst of “Leader epoch mismatch” / FENCED_LEADER_EPOCH as the old epoch’s requests are rejected, followed by brokers refreshing to the new epoch and the warnings stopping.

Diagnostic Commands

All read-only.

# Current leader and epoch — the authoritative epoch everyone must match
kafka-metadata-quorum.sh --bootstrap-controller c1:9093 describe --status

# Per-voter epochs/offsets; spot a node stuck on an old epoch
kafka-metadata-quorum.sh --bootstrap-controller c1:9093 describe --replication

# Count epoch-mismatch events over time — transient vs persistent?
grep -iE 'leader epoch mismatch|FENCED_LEADER_EPOCH|truncate to diverging' \
  /var/log/kafka/controller.log | tail -60

# Are elections churning the epoch (root cause if mismatches persist)?
grep -iE 'election|transition to (Leader|Candidate)|epoch ' \
  /var/log/kafka/controller.log | tail -40

# Inspect leaderEpoch transitions recorded in the metadata log (read-only)
kafka-dump-log.sh --cluster-metadata-decoder \
  --files /var/lib/kafka/__cluster_metadata-0/*.log | grep -i 'LeaderChange\|epoch' | tail -30

# Clock and GC on the suspect node
journalctl -u kafka --since "20 min ago" | grep -iE 'gc|pause'
timedatectl status | grep -i 'synchronized'

If describe --status shows a single stable leader and the mismatch warnings have stopped, the fencing was the normal post-election transient. If one voter in describe --replication shows an older epoch that never advances, that node is stuck.

Step-by-Step Resolution

1. Decide whether it is transient or persistent. A short burst of mismatches right after an election that then stops is healthy — no action needed. Persistent mismatches need investigation.
2. Identify the stuck node via describe --replication: the voter whose epoch trails the leader and does not advance, or the broker repeatedly logging FENCED_LEADER_EPOCH.
3. Check for a zombie/partitioned leader. If a former leader is isolated and still acting on the old epoch, restore its network so it observes the new leader, or restart it to force a fresh fetch.
4. Stabilize leadership if elections are churning. Repeated epoch bumps mean the real problem is election instability — resolve the underlying quorum/network issue so the epoch stops advancing.
5. Let divergent followers truncate. A follower told to “truncate to diverging offset” will roll back its uncommitted tail and resume from the leader automatically; ensure it has connectivity and disk to do so.
6. Verify convergence: describe --status shows one leader and one epoch, describe --replication shows all voters on that epoch with near-zero lag, and the mismatch warnings cease.

Prevention and Best Practices

• Treat a brief post-failover burst of epoch-mismatch warnings as normal; only alert when they persist beyond a short window.
• Keep the controller quorum stable (odd voter count, low-latency network) so elections — and epoch bumps — are rare.
• Keep clocks in sync via NTP/chrony; skew makes nodes lag epoch updates.
• Tune GC/heap on controllers so stop-the-world pauses do not make a node miss leadership changes.
• Ensure brokers can promptly refresh controller leadership (healthy broker-to-controller connectivity) so stale-epoch requests self-correct fast.
• Monitor the epoch from describe --status; a rapidly climbing epoch signals election churn, not just fencing.

Related Errors

• Raft leader election failed — the churn that makes epoch mismatches persist.
• Unable to fetch metadata log — a fenced follower truncating and re-fetching from the new leader.
• Metadata quorum unavailable — when no stable leader exists to define a current epoch at all. See the Kafka guides.

Frequently Asked Questions

Is “Leader epoch mismatch” always a problem? No. After any controller election it appears briefly while stale-epoch requests are fenced and refreshed, then stops. Only persistent mismatches indicate trouble.

What does FENCED_LEADER_EPOCH mean for a broker? The broker sent a request to a controller using an old epoch. It refreshes the current controller/epoch and retries — usually invisible to clients.

What is “truncate to diverging offset”? A follower whose log tail diverged from the new leader must discard its uncommitted tail and resume from the leader’s offset to stay consistent. This is normal Raft recovery.

Why does the epoch keep climbing? Each election increments it. A steadily climbing epoch means repeated elections — fix the election instability, not the mismatch symptom.

Could this cause data loss? Only uncommitted (not-yet-quorum-acknowledged) tail records are truncated, which is correct Raft behavior. Committed metadata is preserved.