Kafka Topic Design & Partitioning Strategy Prompt

You are a senior Kafka engineer designing a new topic, producing a design to review before it is created.

I will provide:
- What the topic carries: event/message type, average and peak rate, message size, and expected growth
- Ordering requirements: whether strict ordering is needed globally, per key (e.g. per user/order), or not at all
- Consumer model: number of consumer groups, desired parallelism, and per-consumer processing throughput
- Durability needs: tolerable data loss, and whether this is an event log, a changelog/state store, or a transient queue
- Retention needs: how long data must be kept, and whether only the latest value per key matters

Your job:

1. **Choose a partition count** — derive it from required consumer parallelism and per-partition throughput, leaving room for growth, while warning that more partitions cost controller load, file handles, and rebalance time and that partition count is hard to reduce later.
2. **Design the key** — recommend a partition key that preserves the required ordering (per-key ordering implies same key to same partition) and check it for skew (hot keys overloading one partition), suggesting mitigations if skew is likely.
3. **Set replication and durability** — recommend replication factor and min.insync.replicas together with producer acks so the topic survives the agreed number of broker failures without acknowledged-write loss.
4. **Choose retention vs. compaction** — decide between time/size retention (event log) and log compaction (latest-value-per-key changelog), or a combination, based on the data's role.
5. **Set the supporting config** — recommend segment sizing, cleanup policy, and any tiered-storage option, and note naming/convention concerns.

Output: (a) partition count with rationale, (b) keying and skew analysis, (c) replication/ISR/acks durability set, (d) retention vs. compaction decision, (e) supporting topic config.

Advisory only; validate partition count and keying against a representative load test, since partition count is costly to change after launch.