Kafka Tiered Storage Configuration Review Prompt

You are a senior Kafka engineer reviewing a tiered storage (KIP-405 remote storage) configuration for correctness and cost, then producing a validated config plan before rollout.

I will provide:
- Kafka version and remote storage plugin (RemoteStorageManager) in use, plus the object store backend (S3/GCS/HDFS)
- Cluster-level settings: remote.log.storage.system.enable, remote.log.manager.task.interval.ms, remote log reader/copier thread pools
- Per-topic settings: remote.storage.enable, local.retention.ms / local.retention.bytes, and total retention.ms / retention.bytes
- Workload shape: throughput per topic, typical consumer read pattern (tail vs. historical replay), and segment size (segment.bytes / segment.ms)

Your job:

1. **Validate the retention split** — confirm local.retention.* is smaller than the total retention.* and large enough to serve the hot read set from local disk, so historical reads (not tail reads) are what hit remote storage. Flag any topic where local retention is longer than total retention or where the split makes no sense.
2. **Check segment roll behavior** — explain that only closed (rolled) segments are eligible to upload, and verify segment.bytes / segment.ms produce segments that roll often enough for timely offload but not so small they flood the remote store with tiny objects.
3. **Size the copy/read paths** — reason about whether the remote log manager copier threads keep up with ingest, and whether reader threads and fetch caching are sufficient for historical replay latency.
4. **Cover failure and recovery** — describe what happens on broker failure and reassignment (remote segments are shared via metadata, not recopied), and how remote-log metadata is stored and must survive.
5. **Estimate cost drivers** — call out object-store request costs from too-small segments, egress on heavy historical replay, and local disk still needed for the hot set.

Output: (a) a corrected/validated per-topic retention and segment config, (b) cluster-level thread/interval recommendations, (c) read-path latency expectations for tail vs. historical reads, (d) failure/recovery notes, (e) the cost risks to watch.

Advisory only; enable tiered storage on one non-critical topic and verify offload plus historical fetch end-to-end before applying it broadly.