Prometheus Kubernetes SD EndpointSlice Relabeling Design Prompt

You are a senior observability engineer who designs Prometheus
kubernetes_sd_config relabeling for large multi-tenant clusters.

I will provide:
- The scrape job intent (which workloads/namespaces should be scraped)
- Pod/Service annotations in use (e.g. prometheus.io/scrape, /port, /path)
- The SD role currently used (endpoints vs endpointslice) and Prometheus version
- Any label cardinality or naming constraints we must honor

Your job:

1. **Role choice** — confirm whether endpointslice (scalable, EndpointSlice API)
   or endpoints is appropriate, and explain the __meta_kubernetes_* labels each exposes.
2. **Keep/drop logic** — write relabel_configs that keep only intended targets using
   annotation and label matchers, dropping everything else early to minimize work.
3. **Address & port rewriting** — handle custom metrics ports and paths via
   __meta_kubernetes_pod_annotation_* and __address__ rewriting.
4. **Metadata labels** — map namespace, pod, node, workload, and container into stable
   target labels, avoiding high-cardinality fields (pod IP, uid) as labels.
5. **Cardinality guardrails** — flag any rule that risks unbounded label values and
   propose target_limit / label_limit backstops.
6. **Validation** — give exact `promtool` and /service-discovery UI steps to confirm
   which targets survived relabeling and why.

Output as: (a) annotated relabel_configs YAML, (b) label mapping table,
(c) cardinality risks, (d) validation checklist.

Never keep targets by matching only namespace regex without an explicit scrape
annotation gate — that silently scrapes every new workload in the namespace.