Kafka Broker Rolling Restart Runbook Prompt

You are a senior Kafka operator writing a step-by-step rolling-restart runbook for a production cluster. The goal is zero client-visible impact: no partition goes offline, no topic drops below its minimum in-sync replica count, and leadership churn is minimized.

I will provide:
- Cluster topology: broker count, KRaft vs. ZooKeeper mode, which nodes are controllers, and rack/AZ assignments
- Replication settings: default.replication.factor, min.insync.replicas, and any per-topic overrides
- The reason for the restart: config change, JVM/version upgrade, cert rotation, or host maintenance
- Current health: under-replicated partition count, offline partitions, and any brokers already lagging

Your job:

1. **Gate on pre-flight health** — require zero under-replicated and zero offline partitions before starting, and verify every partition has full ISR so losing one broker still satisfies min.insync.replicas. Give the exact commands (kafka-topics.sh --describe --under-replicated-partitions, --unavailable-partitions).
2. **Order the brokers** — restart controllers appropriately for the mode (in KRaft, restart non-active controllers first, then the active controller last after confirming quorum; in ZooKeeper mode, restart the controller broker last). Sequence remaining brokers one AZ at a time, never two replicas of the same partition together.
3. **Use controlled shutdown** — confirm controlled.shutdown.enable is true so leadership is migrated gracefully, and describe how to move leadership proactively if needed.
4. **Define the per-broker wait gate** — after each restart, block until under-replicated partitions return to zero and ISR is fully caught up before touching the next broker; specify the metric/command to poll and a timeout.
5. **Plan rollback and abort criteria** — state the conditions (offline partitions, ISR not recovering within the timeout, controller quorum loss) that halt the rollout, and the recovery steps.

Output: (a) pre-flight checklist with commands, (b) the exact broker restart order with justification, (c) the per-broker wait/verify gate, (d) abort and rollback criteria, (e) a post-restart validation step.

Advisory only; dry-run the ordering against your inventory and restart one broker in a maintenance window before automating the full loop.