🚨 Kafka Error Guide: 'ProducerFencedException' — Fixing Fenced Transactional Producers

Overview

A transactional Kafka producer throws a fatal ProducerFencedException and can no longer produce. The transaction coordinator has fenced this producer because a newer producer instance registered the same transactional.id with a higher epoch:

org.apache.kafka.common.errors.ProducerFencedException: There is a newer producer with the same transactionalId which fences the current one.

Kafka Streams surfaces the same condition as a task-fencing error that shuts the thread down:

ERROR stream-thread [app-StreamThread-2] Encountered ProducerFencedException; it means all tasks belonging to this thread have been fenced; the thread will now rejoin the group (org.apache.kafka.streams.processor.internals.StreamThread)

This is a fatal, non-retriable error for the affected producer instance: the current KafkaProducer object must be closed and recreated; retrying on the same instance will keep failing.

Symptoms

• A transactional producer fails commitTransaction()/send() with ProducerFencedException and stops producing.
• Two or more processes or pods appear to share one transactional.id, and they intermittently fence each other.
• A producer that paused (long GC, network stall, or max.poll.interval.ms breach) resumes and is immediately fenced.
• Kafka Streams threads repeatedly die and rejoin with ProducerFencedException under exactly_once_v2.
• After a rolling restart or a failed instance being replaced, the old instance — if still alive — is fenced by its successor.

Common Root Causes

• Duplicate transactional.id across live instances. Two application instances configured with the same static transactional.id (e.g. hardcoded, or derived from a value that is not unique per instance) register with the coordinator; the later one wins and fences the earlier.
• Zombie producer after a stall. A producer paused long enough that its transaction.timeout.ms expired or that its group considered it dead; a replacement took over the transactional.id, and the revived original is now a zombie that gets fenced — this is the fencing mechanism working as designed.
• Overlapping old and new pods on deploy. During a rolling deployment the new pod starts with the same transactional.id before the old pod fully shuts down.
• transaction.timeout.ms too low for the work. The transaction outlives the configured timeout, the coordinator aborts it, and the next commit is fenced.
• Kafka Streams instances sharing state incorrectly. Misconfigured application.id/group.instance.id causing two threads to claim the same task’s transactional identity.
• Manual producer restart without closing the old one. A new KafkaProducer with the same transactional.id is created while the previous object is still open.

Diagnostic Workflow

1. Confirm it is a fencing, not a timeout. Read the stack trace — ProducerFencedException specifically means a newer producer took the ID. Distinguish it from InvalidProducerEpochException or a plain TimeoutException.

2. Identify the transactional.id in use. Find the exact transactional.id each instance sets. If it is static, that is the prime suspect for duplication.

3. List active transactions on the cluster. Inspect what the coordinator currently tracks:

kafka-transactions.sh --bootstrap-server broker:9092 list

Then describe the suspect ID to see its state, producer id, and epoch:

kafka-transactions.sh --bootstrap-server broker:9092 describe --transactional-id my-app-tx-1

4. Look for long-running or stuck transactions. Find transactions open longer than expected, which point at a timeout or a hung instance:

kafka-transactions.sh --bootstrap-server broker:9092 find-hanging --broker-id 1

5. Correlate with deploys and pauses. Check whether fencing timestamps line up with a rolling deploy, a GC pause, or a network blip in the application logs — a revived zombie being fenced is expected behavior.

6. Verify producer configuration. Confirm enable.idempotence=true, a sane transaction.timeout.ms (and that it does not exceed the broker’s transaction.max.timeout.ms), and that transactional.id is unique per logical producer.

Example Root Cause Analysis

A payments service ran three replicas, each configured with transactional.id=payments-writer. Under normal load only one replica produced at a time behind a leader election, so the shared ID went unnoticed. During a traffic spike the standby replicas briefly began producing too; the coordinator saw three registrations of payments-writer, granted the newest one a higher epoch, and fenced the other two with ProducerFencedException.

kafka-transactions.sh describe --transactional-id payments-writer showed the producer epoch had jumped several times in minutes — a clear sign of repeated re-registration by different instances. The root cause was not a Kafka bug but a configuration error: transactional.id must be stable per logical producer and unique across concurrently active instances.

The fix was to derive transactional.id from a stable, per-instance identity (payments-writer-<ordinal> from the StatefulSet pod ordinal), so each replica owned a distinct transactional identity. After the change, epochs stabilized, fencing stopped, and exactly-once semantics held. The application also added handling to close and recreate the producer on any ProducerFencedException rather than retrying the fenced instance.

Prevention Best Practices

• Make transactional.id unique per active instance. Derive it from a stable per-pod identity (StatefulSet ordinal, host id) so two live instances never share one ID.
• Treat fencing as fatal in code. On ProducerFencedException, close the producer and recreate it (or let the framework restart the task) — never retry the fenced object.
• Size transaction.timeout.ms to the real work. Set it above your worst-case transaction duration but within the broker’s transaction.max.timeout.ms.
• Serialize producers during deploys. Ensure the old instance releases the transactional.id (graceful shutdown, close()) before the replacement starts producing.
• Prefer exactly_once_v2 in Streams. It reduces the number of producers and is more resilient to fencing than the older EOS mode.
• Monitor producer epochs and hanging transactions. Alert on rising epoch churn and on transactions flagged by find-hanging.

Quick Command Reference

# List all transactional IDs the coordinator is tracking
kafka-transactions.sh --bootstrap-server broker:9092 list

# Describe one transactional ID (state, producer id, epoch)
kafka-transactions.sh --bootstrap-server broker:9092 describe --transactional-id my-app-tx-1

# Find hanging/stuck transactions on a broker
kafka-transactions.sh --bootstrap-server broker:9092 find-hanging --broker-id 1

# Check the broker-side cap on transaction timeout
kafka-configs.sh --bootstrap-server broker:9092 --describe --entity-type brokers --entity-default | grep transaction.max.timeout.ms

# Inspect the transaction state topic health
kafka-topics.sh --bootstrap-server broker:9092 --describe --topic __transaction_state

Conclusion

ProducerFencedException is Kafka’s exactly-once machinery doing exactly what it should: when a newer producer registers the same transactional.id, the coordinator fences the older one to guarantee only one producer with that identity can commit. The error is almost never a broker fault — it points at a duplicated transactional.id across live instances, a zombie producer revived after a stall, or overlapping instances during a deploy. Fix it by giving each concurrently active producer a unique, stable transactional identity, sizing transaction.timeout.ms correctly, and treating the exception as fatal by recreating the producer rather than retrying the fenced one.