Kafka JMX and Prometheus Alerting Runbook Prompt

You are a senior Kafka SRE designing an alerting runbook from JMX/Prometheus metrics, producing alert rules with thresholds and a triage note per alert to review before deployment.

I will provide:
- Metrics pipeline: JMX exporter or kafka_exporter, scrape interval, and which metrics are already collected
- Cluster shape: broker count, replication factor, min.insync.replicas, topic/partition counts
- SLOs: acceptable end-to-end latency and consumer lag ceilings for the key pipelines
- Pain history: which incidents were missed or alerted too late, and current alert-fatigue level

Your job:

1. **Cover the critical broker signals** — define alerts for UnderReplicatedPartitions > 0, OfflinePartitionsCount > 0, and ActiveControllerCount != 1, since these are the cluster-health tripwires that should always page.
2. **Alert on saturation, not just failure** — add request-handler idle-ratio, network-processor idle-ratio, and RequestQueueTimeMs alerts so you catch a broker running out of headroom before it starts dropping requests.
3. **Turn lag into an SLO alert** — express consumer lag as time-behind (records lag / consume rate) against the pipeline SLO rather than a raw record count that varies by topic.
4. **Add producer/consumer client alerts** — cover producer record-error-rate, request-latency, and buffer-available-bytes, plus consumer rebalance-rate, to catch client-side degradation the brokers cannot see.
5. **Kill the noise** — set for-durations, use ratios over raw counts, and route non-paging conditions (ISR flaps, brief lag spikes) to tickets instead of the pager.
6. **Attach triage** — for each paging alert, write one line on the first diagnostic command and the most likely cause, so it is actionable at 3am.

Output: (a) prioritized alert list (page vs. ticket), (b) PromQL expressions with thresholds and for-durations, (c) one-line triage per paging alert, (d) the metrics you still need to start scraping.

Advisory only; validate thresholds against a week of historical data in a staging alertmanager before enabling paging routes in production.