🚨 Kafka Error Guide: 'Node 1 disconnected' Established Broker Connection Dropped

Exact Error Message

This appears in a Kafka client log after metadata has loaded and a connection to a known, numbered broker drops:

[2026-06-29 16:05:33,712] INFO [Consumer clientId=consumer-4 groupId=billing] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2026-06-29 16:05:33,713] WARN [Consumer clientId=consumer-4 groupId=billing] Connection to node 1 (kafka-1.internal/10.0.4.31:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Common close variants point at the same condition:

[2026-06-29 16:05:31,500] WARN [Consumer clientId=consumer-4] Connection to node 1 (kafka-1.internal/10.0.4.31:9092) terminated during authentication.
[2026-06-29 16:05:31,510] WARN [Consumer clientId=consumer-4] Received failed for node 1
[2026-06-29 16:05:31,512] WARN [NetworkClient] Disconnected while requesting ApiVersion(s) from node 1; it is likely that the node is not available, or the broker version is incompatible.

You may also see Send failed for node 1 for the write direction.

What the Error Means

Unlike node -1 (the bootstrap placeholder), node 1 is a real broker — the client already loaded cluster metadata and knows broker ID 1 lives at a specific address. “Node 1 disconnected” means an established connection to that broker was closed. The client logs the disconnect, then retries to re-establish it.

This is normal, low-level connection lifecycle messaging in many cases: Kafka opens connections lazily and reaps idle ones, so an occasional Node N disconnected at INFO level during quiet periods is benign. It becomes a problem when it is frequent, sustained, or paired with Disconnected while requesting ApiVersion(s), which signals the connection died during the initial API-version negotiation — typically a broker that is down, incompatible, or being spoken to with the wrong protocol.

So the distinction matters: this is post-metadata (you know it is broker 1), the connection existed, and the close is being reported at the Kafka layer rather than as a raw IOException.

Common Causes

• connections.max.idle.ms reaping. The broker (default 10 min) closes idle connections; the client logs Node N disconnected and reconnects on next use. Benign.
• Broker restart, crash, or rolling deploy. Broker 1 went down or is restarting; connections drop until it returns or the partition leader moves.
• Version / protocol incompatibility. Disconnected while requesting ApiVersion(s) means the client and broker could not agree on API versions — an old broker with a newer client, or a non-Kafka service on the port.
• Plaintext/SSL mismatch causing the connection to die during early negotiation (“terminated during authentication”).
• Network instability or middlebox idle reset dropping the flow.
• Broker overload shedding connections under load (Receive failed / Send failed).

How to Reproduce the Error

Run an idle consumer past the broker idle timeout, then watch it reconnect:

kafka-console-consumer.sh --bootstrap-server kafka-1.internal:9092 --topic billing --group billing
# leave it idle beyond connections.max.idle.ms; the client logs Node N disconnected, then reconnects on activity

For the ApiVersion variant, point a current client at a service that is not a compatible Kafka broker (or a very old broker) on 9092 — the client logs Disconnected while requesting ApiVersion(s).

Diagnostic Commands

All read-only.

Confirm broker 1 is up and recently stable:

sudo systemctl status kafka --no-pager | grep -E 'Active|since'
sudo journalctl -u kafka --no-pager --since "20 min ago" | grep -iE 'shutdown|started|controlled.shutdown|OutOfMemory'

Negotiate API versions directly — this is exactly what “ApiVersion(s)” failures hit:

kafka-broker-api-versions.sh --bootstrap-server kafka-1.internal:9092

Check the broker’s idle setting to explain periodic disconnects:

grep -E 'connections.max.idle.ms' /opt/kafka/config/server.properties

Verify reachability and whether the port speaks TLS:

nc -z -v -w 5 kafka-1.internal 9092
openssl s_client -connect kafka-1.internal:9093 -brief </dev/null

Observe connection lifetime to detect a periodic reaper/middlebox:

ss -tni dst 10.0.4.31:9092

Step-by-Step Resolution

1. Decide if it is benign. Occasional Node N disconnected at INFO during idle periods, with prompt reconnects and no application impact, needs no action — it is idle reaping. Only act on frequent/sustained disconnects or consumer rebalancing.

2. Check broker health. If broker 1 restarted or crashed (systemctl, journalctl), that explains the drop. Ensure clients have retries and that partition leadership fails over; fix the restart/OOM cause.

3. For ApiVersion failures, check compatibility. Disconnected while requesting ApiVersion(s) means negotiation failed. Run kafka-broker-api-versions.sh. If it errors, the broker is down, far older than the client, or the port is not a real Kafka broker. Align broker/client versions or fix the target.

4. Rule out protocol mismatch. “Terminated during authentication” plus immediate disconnects suggests plaintext vs SSL. Probe with openssl s_client and set the client’s security.protocol to match the listener.

5. Address idle/middlebox drops. If disconnects recur on a fixed interval, align client connections.max.idle.ms below the broker’s and any load balancer/firewall idle timeout so reconnects are orderly.

6. Check load. Sustained Receive failed/Send failed for a node can indicate broker overload; review broker request-queue and network-thread metrics and scale if needed.

Prevention and Best Practices

• Keep client and broker versions within a supported compatibility range to avoid ApiVersion(s) negotiation failures.
• Set client connections.max.idle.ms lower than the broker’s and any middlebox idle timeout so idle reaping is predictable and quiet.
• Standardize security.protocol per listener and validate it so a plaintext client never hits an SSL port.
• Use controlled.shutdown.enable=true with rolling restarts so leadership fails over and clients reconnect cleanly.
• Treat occasional idle disconnects as normal; alert on disconnect rate and on consumer-group rebalances, not on individual INFO lines.
• The free incident assistant can correlate a flood of Node N disconnected lines with broker status to pinpoint whether it is a deploy, a version mismatch, or overload.

Related Errors

• Connection to node -1 could not be established — the bootstrap failure before metadata; node -1 vs node N is the key difference.
• Connection reset by peer — a raw RST on an established connection, often the transport-level cause behind a disconnect.
• Broken pipe — write-side failure when sending to a node that already closed.
• SocketTimeoutException — a connect/request timeout rather than a reported disconnect.

Frequently Asked Questions

Is Node 1 disconnected an error? Often not. Kafka reaps idle connections and logs this at INFO, then reconnects on demand. It matters when it is frequent, sustained, or causes consumer rebalances.

How is node 1 different from node -1? Node -1 is the synthetic bootstrap connection used before metadata loads. Node 1 is a real broker the client already discovered. Node N disconnects are post-metadata.

What does “Disconnected while requesting ApiVersion(s)” mean? The connection died during API-version negotiation — usually the broker is down, the broker is far older than the client, or the port is not a compatible Kafka broker.

Why do I see this every ten minutes? That matches the default connections.max.idle.ms (10 min). The broker is reaping idle connections. Lower the client idle setting to make recycling proactive.

Could a load balancer cause periodic node disconnects? Yes. An LB or firewall idle timeout will drop the flow on a schedule. Align the client idle timeout below the device’s.