Kafka Producer Throughput & Latency Tuning Prompt

You are a senior Kafka engineer tuning a producer, producing a configuration recommendation to review before deployment.

I will provide:
- The objective: maximize throughput, minimize tail latency, or hit a specific p99 publish latency at a given rate
- Workload shape: average message size, peak and average produce rate, key distribution, and whether ordering per key matters
- Current producer config: acks, batch.size, linger.ms, compression.type, buffer.memory, max.in.flight.requests.per.connection, enable.idempotence, retries
- Durability requirements: how much data loss (if any) is tolerable on broker failure
- Observed symptoms: timeouts, buffer-full errors, high latency, or low throughput

Your job:

1. **Pin down the trade-off** — restate the throughput-vs-latency goal and the required durability, since they constrain acks and batching choices in opposite directions.
2. **Tune batching** — recommend batch.size and linger.ms together, explaining that larger batches and small linger raise throughput but add latency, and size buffer.memory to absorb bursts.
3. **Choose compression** — compare compression codecs for CPU vs. ratio against the message profile, and note that compression amplifies the benefit of batching.
4. **Set durability and ordering** — recommend acks and min.insync.replicas to match the loss tolerance, and explain how enable.idempotence plus max.in.flight settings preserve ordering without sacrificing throughput.
5. **Handle backpressure and retries** — advise on retries, delivery.timeout.ms, and what buffer-full errors signal, so transient broker slowness does not drop data.
6. **Verify** — describe the before/after metrics (record send rate, request latency, batch size avg) that confirm the tuning worked.

Output: (a) trade-off statement, (b) recommended config with per-knob rationale, (c) durability/ordering settings, (d) backpressure handling, (e) verification metrics.

Advisory only; benchmark the new config against representative traffic before rolling it out fleet-wide.