🚨 Telegraf Error Guide: 'outputs.kafka circuit breaker is open' — Fix Kafka Write Failures

Overview

The Kafka output uses the Sarama client, whose circuit breaker trips after repeated produce failures to a broker. Telegraf then logs:

E! [outputs.kafka] Could not write to Kafka: circuit breaker is open

The breaker is a symptom: Sarama opened it because prior produce requests kept failing (broker down, wrong address, auth rejected, or no leader). While it is open, Telegraf stops attempting writes to that broker and buffers metrics until the breaker half-opens and a probe succeeds.

Symptoms

• Could not write to Kafka: circuit breaker is open repeating each flush.
• Often preceded by dial tcp ... connection refused, EOF, or kafka: client has run out of available brokers.
• Consumers downstream see a gap in the topic.
• Telegraf’s buffer grows and eventually logs metric buffer overflow.

Common Root Causes

• One or more brokers unreachable — down, restarting, or network-partitioned.
• Wrong brokers list — advertised listener addresses that Telegraf cannot resolve or route to.
• TLS/SASL misconfiguration — handshake or auth failing so every produce is rejected.
• Leader election in progress — no partition leader available during a broker restart or rebalance.
• required_acks = -1 (all) with an under-replicated partition where not enough in-sync replicas exist to acknowledge.
• Topic does not exist and auto-create is disabled on the cluster.

Diagnostic Workflow

Find the failure that tripped the breaker — the breaker message alone is not enough:

journalctl -u telegraf --since '15 min ago' \
  | grep -iE 'kafka|circuit breaker|broker|EOF|refused'

Confirm the plugin’s broker list and auth settings:

[[outputs.kafka]]
  brokers = ["kafka-1:9092", "kafka-2:9092", "kafka-3:9092"]
  topic = "telegraf-metrics"
  required_acks = 1
  enable_tls = true
  sasl_username = "${KAFKA_USER}"
  sasl_password = "${KAFKA_PASSWORD}"

Test raw reachability to every broker from Telegraf’s host:

for b in kafka-1 kafka-2 kafka-3; do nc -vz "$b" 9092; done

Verify broker metadata, topic existence, and leadership with the Kafka tooling:

kafka-broker-api-versions.sh --bootstrap-server kafka-1:9092
kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic telegraf-metrics

Look at the Leader: and Isr: fields; a Leader: none or shrunken ISR explains produce failures under required_acks = -1. Once brokers are healthy, the breaker half-opens automatically and buffered metrics flush.

Example Root Cause Analysis

After a rolling Kafka upgrade, Telegraf logged circuit breaker is open continuously even though nc -vz connected to all three brokers. kafka-topics.sh --describe showed the telegraf-metrics topic had Isr: 1 (only one in-sync replica) while the plugin used required_acks = -1, which requires all in-sync replicas — but min.insync.replicas=2 meant produces were rejected with NOT_ENOUGH_REPLICAS, tripping the breaker. The immediate fix was to restore the second replica (an unclean broker had not rejoined the ISR); a follow-up hardening changed required_acks to 1 for metrics, where a single-ack durability tradeoff is acceptable, so a single lagging replica never again halts telemetry.

Prevention Best Practices

• List all brokers in brokers so the client can fail over when one is down.
• Match required_acks to your durability needs; for high-volume metrics, 1 avoids blocking on ISR shrinkage that -1 is sensitive to.
• Use advertised listener addresses that Telegraf can actually resolve and route to, especially across networks/VPCs.
• Pre-create the topic (or enable auto-create) with a replication factor and min.insync.replicas you can reliably meet.
• Size metric_buffer_limit to ride out broker restarts, and add disk buffering for longer Kafka outages.
• Alert on the Telegraf Kafka output error rate and on Kafka under-replicated partitions.

Quick Command Reference

# Find the failure that tripped the breaker
journalctl -u telegraf --since '15 min ago' | grep -iE 'kafka|circuit breaker'

# Reach every broker
for b in kafka-1 kafka-2 kafka-3; do nc -vz "$b" 9092; done

# Inspect topic leadership and ISR
kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic telegraf-metrics

# Check broker availability
kafka-broker-api-versions.sh --bootstrap-server kafka-1:9092

# Watch for the error live
journalctl -u telegraf -f | grep -i 'circuit breaker'

Conclusion

outputs.kafka circuit breaker is open is Sarama protecting itself after repeated produce failures — the real cause is an unreachable broker, an auth/TLS problem, a missing topic, or an ISR too small for your required_acks. Read the errors that preceded the breaker, test every broker with nc, and inspect leadership with kafka-topics.sh --describe. Right-size required_acks and buffer limits so a single broker hiccup never halts telemetry. If metrics drop while the breaker is open, see the metric buffer overflow guide. More fixes in the Telegraf guides.