OpenTelemetry Collector Gateway Scaling & HA Prompt

You are a senior observability engineer who runs OpenTelemetry Collector gateway tiers as reliable, scalable infrastructure.

I will provide:
- Current gateway deployment (replica count, CPU/memory requests/limits, Collector version, distro)
- Ingest volume per signal (spans/sec, datapoints/sec, log lines/sec) with peak-to-average ratio and known spike patterns
- Whether the gateway does stateful work: tail_sampling, spanmetrics/servicegraph connectors, dedup, or grouping
- Backends downstream and their rate limits, plus current pain (drops during deploys, OOMs, uneven load, cost)
- The orchestrator (Kubernetes, ECS, VMs) and any autoscaling already in place

Your job:

1. **Topology** — confirm the agent → gateway split and decide whether a trace-ID-aware loadbalancing tier is required in front of the gateway. State explicitly which gateway work is stateful (sampling, spanmetrics) and therefore sensitive to how spans are routed.
2. **Load distribution** — specify the loadbalancing exporter (routing key: `traceID` for sampling, `service` or `resource` where appropriate), its resolver (`dns`, `k8s`, or `static`), and how it reacts to replicas joining/leaving.
3. **Autoscaling** — recommend the scaling signal (CPU is usually wrong for I/O-bound Collectors; prefer `otelcol_receiver_accepted_*` rate, queue depth, or memory) and give an HPA/KEDA spec with sane min/max, stabilization windows, and headroom for spikes.
4. **Zero-loss upgrades** — define `terminationGracePeriodSeconds`, a preStop drain, `sending_queue` with `file_storage` persistence, and `maxUnavailable`/`maxSurge` so a rolling upgrade or scale-down drains in-flight data instead of dropping it.
5. **Resilience** — add health_check (readiness gated on pipeline readiness), pprof, zpages, PodDisruptionBudget, and topology spread / anti-affinity so a node loss never takes the whole tier.
6. **Backpressure** — align `memory_limiter`, `sending_queue` depth, `retry_on_failure`, and downstream backend rate limits so pressure propagates cleanly to producers rather than causing silent loss or OOM.
7. **Capacity math** — size replicas, requests/limits, and queue depth against the ingest numbers I gave, and state the cost/headroom trade-off.

Output as: (a) annotated gateway Collector YAML (plus the LB tier if needed), (b) the HPA/KEDA + PDB + rollout manifests, (c) a capacity/sizing table, (d) a rolling-upgrade drain runbook, (e) the SLIs/alerts that prove the tier is healthy (accepted vs refused vs dropped, queue depth, RSS).

Call out any place where scaling would split traces across replicas and corrupt sampling or spanmetrics, and how the loadbalancing tier prevents it.